Security, privacy and governance by design
Designing AI systems that protect data and stand up to scrutiny: mapping data flows, identity and access, retention and residency, vendor assessment, an AI system inventory, risk classification, human oversight, documentation, and frameworks such as the NIST AI RMF and ISO/IEC 42001.
๐ฏ By the end of this lesson you will be able to:
- Map an AI system's data flows and apply privacy and security controls to each
- Set up practical AI governance: inventory, risk classification, oversight and documentation
- Know the main frameworks and regulations that shape responsible AI
In this lesson
Why "by design" matters
Security, privacy and governance are cheap to build in and expensive to add later. An assistant that already sends every document to an external model without permission checks, logs personal data forever and has no owner is hard to fix once people depend on it. Architects make these choices at the start, so teams can move fast safely.
Map the data flows
For each AI feature, draw where data goes: from users, documents and databases, into prompts, to which model provider and region, into logs, traces, caches and evaluation datasets, and back to users. Then classify the data at each step (Lesson 6) and apply controls.
| Control | What it means in practice |
|---|---|
| Minimisation | Send only the data the task needs; mask or drop personal fields where possible |
| Identity and access | The AI acts with the user's permissions, not a super-user account; retrieval and tools are filtered per user |
| Residency | Data is processed and stored in approved regions |
| Retention | Prompts, outputs, logs and memory are kept only as long as needed, then deleted |
| Encryption | In transit and at rest, including vector stores, caches and logs |
| Audit | Who asked what, which data was used and what the AI did, available for investigation |
| Vendor terms | Contracts cover data use for training, retention, sub-processors and security standards |
Practical AI governance
- Inventory: keep a register of every AI system and use, including vendor tools, with an owner, purpose, data used, model and users.
- Risk classification: rate each use. Uses affecting people's rights, safety, jobs, credit or health are high risk and need the strongest controls.
- Proportionate controls: low-risk uses get light checks; high-risk uses need evaluation evidence, human oversight, documentation and formal approval.
- Human oversight: decide where people review, approve or can override AI outputs, and make sure they have time and authority to do so.
- Transparency: tell people when they are interacting with AI or receiving AI-generated content where it matters.
- Monitoring and review: re-check high-risk systems regularly and when models, data or purpose change.
Documentation that helps
- System card: purpose, users, model and version, data sources, known limitations, evaluation results and owners.
- Risk assessment: what could go wrong, for whom, and the controls in place.
- Decision log: key design choices and approvals (ADRs from Lesson 24).
- User-facing notice: plain-language explanation of what the AI does and its limits.
Frameworks and regulations to know
| Name | What it is |
|---|---|
| NIST AI Risk Management Framework | A voluntary US framework organised around four functions: Govern, Map, Measure and Manage |
| ISO/IEC 42001 | An international standard for an AI management system, which organisations can be certified against |
| EU AI Act | EU law taking a risk-based approach, with the strictest obligations for high-risk uses and some uses prohibited; obligations apply in phases |
| Data protection laws (e.g. GDPR) | Rules on personal data that apply to AI systems just like any other processing |
| OWASP Top 10 for LLM Applications | Community list of the main security risks for LLM apps (Lesson 17) |
What this means for your role
Developer
Implement user-scoped access, data minimisation, retention jobs and audit logging as standard platform features, not per-feature extras.
Tester / QA
Test the controls: permission filtering, data masking, retention deletion and audit completeness, alongside quality.
Business Analyst / PO
Capture data use, purpose, affected people and oversight requirements as part of every AI feature's requirements.
Tech Lead / Architect
Own the data-flow maps, control set and system cards, and make the AI inventory and risk classification part of the architecture review.
Project / Delivery Manager
Plan time for risk assessment, documentation and approvals, scaled to the risk level, and keep the AI inventory up to date.
Scrum Master / Agile
Make governance steps visible in the workflow, such as a risk check before build and a review before launch, so they don't become last-minute blockers.
Try it: write a one-page system card
- Pick an AI feature or tool your team uses.
- Write its purpose, users, model, data sources and where data is processed and stored.
- Add its risk level, known limitations, evaluation evidence, human oversight points and owner. Note anything you couldn't answer, as those are your governance gaps.
Key takeaways
- Design security, privacy and governance in from the start; retrofitting is expensive.
- Map data flows and apply minimisation, user-scoped access, residency, retention, encryption, audit and vendor controls.
- Practical governance: inventory, risk classification, proportionate controls, human oversight, transparency and regular review.
- Know frameworks such as NIST AI RMF, ISO/IEC 42001 and the EU AI Act, and involve legal and compliance early.
Key terms
- Data minimisation
- Using only the personal or sensitive data that a task actually needs.
- Data residency
- Requirements about the countries or regions where data is processed and stored.
- AI inventory
- A register of an organisation's AI systems and uses, with owners and risk levels.
- System card
- A document describing an AI system's purpose, data, model, limits, evaluation and ownership.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. With whose permissions should an AI assistant retrieve documents?
2. What is the first step in practical AI governance?
3. Which AI use should usually be classified as high risk?
4. The NIST AI Risk Management Framework is organised around which four functions?
5. Why should governance offer a fast track for low-risk uses?