Nishaglobal Education logo

Security, privacy and governance by design

Designing AI systems that protect data and stand up to scrutiny: mapping data flows, identity and access, retention and residency, vendor assessment, an AI system inventory, risk classification, human oversight, documentation, and frameworks such as the NIST AI RMF and ISO/IEC 42001.

Lesson 26 of 4418 min readQuiz: 5 questions

๐ŸŽฏ By the end of this lesson you will be able to:

  • Map an AI system's data flows and apply privacy and security controls to each
  • Set up practical AI governance: inventory, risk classification, oversight and documentation
  • Know the main frameworks and regulations that shape responsible AI
In this lesson
  1. Why "by design" matters
  2. Map the data flows
  3. Practical AI governance
  4. Documentation that helps
  5. Frameworks and regulations to know
  6. What this means for your role
  7. Key takeaways
  8. Quiz

Why "by design" matters

Security, privacy and governance are cheap to build in and expensive to add later. An assistant that already sends every document to an external model without permission checks, logs personal data forever and has no owner is hard to fix once people depend on it. Architects make these choices at the start, so teams can move fast safely.

Map the data flows

For each AI feature, draw where data goes: from users, documents and databases, into prompts, to which model provider and region, into logs, traces, caches and evaluation datasets, and back to users. Then classify the data at each step (Lesson 6) and apply controls.

ControlWhat it means in practice
MinimisationSend only the data the task needs; mask or drop personal fields where possible
Identity and accessThe AI acts with the user's permissions, not a super-user account; retrieval and tools are filtered per user
ResidencyData is processed and stored in approved regions
RetentionPrompts, outputs, logs and memory are kept only as long as needed, then deleted
EncryptionIn transit and at rest, including vector stores, caches and logs
AuditWho asked what, which data was used and what the AI did, available for investigation
Vendor termsContracts cover data use for training, retention, sub-processors and security standards

Practical AI governance

AI systemwith an owner1. Inventoryregister every use2. Classify risklow ยท medium ยท high3. Apply controlsproportionate to risk4. Monitor + reviewwhen anything changes
A lightweight governance loop: know your AI systems, classify their risk, apply proportionate controls, monitor them and review regularly.
  1. Inventory: keep a register of every AI system and use, including vendor tools, with an owner, purpose, data used, model and users.
  2. Risk classification: rate each use. Uses affecting people's rights, safety, jobs, credit or health are high risk and need the strongest controls.
  3. Proportionate controls: low-risk uses get light checks; high-risk uses need evaluation evidence, human oversight, documentation and formal approval.
  4. Human oversight: decide where people review, approve or can override AI outputs, and make sure they have time and authority to do so.
  5. Transparency: tell people when they are interacting with AI or receiving AI-generated content where it matters.
  6. Monitoring and review: re-check high-risk systems regularly and when models, data or purpose change.

Documentation that helps

  • System card: purpose, users, model and version, data sources, known limitations, evaluation results and owners.
  • Risk assessment: what could go wrong, for whom, and the controls in place.
  • Decision log: key design choices and approvals (ADRs from Lesson 24).
  • User-facing notice: plain-language explanation of what the AI does and its limits.

Frameworks and regulations to know

NameWhat it is
NIST AI Risk Management FrameworkA voluntary US framework organised around four functions: Govern, Map, Measure and Manage
ISO/IEC 42001An international standard for an AI management system, which organisations can be certified against
EU AI ActEU law taking a risk-based approach, with the strictest obligations for high-risk uses and some uses prohibited; obligations apply in phases
Data protection laws (e.g. GDPR)Rules on personal data that apply to AI systems just like any other processing
OWASP Top 10 for LLM ApplicationsCommunity list of the main security risks for LLM apps (Lesson 17)

What this means for your role

Developer

Implement user-scoped access, data minimisation, retention jobs and audit logging as standard platform features, not per-feature extras.

Tester / QA

Test the controls: permission filtering, data masking, retention deletion and audit completeness, alongside quality.

Business Analyst / PO

Capture data use, purpose, affected people and oversight requirements as part of every AI feature's requirements.

Tech Lead / Architect

Own the data-flow maps, control set and system cards, and make the AI inventory and risk classification part of the architecture review.

Project / Delivery Manager

Plan time for risk assessment, documentation and approvals, scaled to the risk level, and keep the AI inventory up to date.

Scrum Master / Agile

Make governance steps visible in the workflow, such as a risk check before build and a review before launch, so they don't become last-minute blockers.

Try it: write a one-page system card

  1. Pick an AI feature or tool your team uses.
  2. Write its purpose, users, model, data sources and where data is processed and stored.
  3. Add its risk level, known limitations, evaluation evidence, human oversight points and owner. Note anything you couldn't answer, as those are your governance gaps.

Key takeaways

  • Design security, privacy and governance in from the start; retrofitting is expensive.
  • Map data flows and apply minimisation, user-scoped access, residency, retention, encryption, audit and vendor controls.
  • Practical governance: inventory, risk classification, proportionate controls, human oversight, transparency and regular review.
  • Know frameworks such as NIST AI RMF, ISO/IEC 42001 and the EU AI Act, and involve legal and compliance early.

Key terms

Data minimisation
Using only the personal or sensitive data that a task actually needs.
Data residency
Requirements about the countries or regions where data is processed and stored.
AI inventory
A register of an organisation's AI systems and uses, with owners and risk levels.
System card
A document describing an AI system's purpose, data, model, limits, evaluation and ownership.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. With whose permissions should an AI assistant retrieve documents?

  2. 2. What is the first step in practical AI governance?

  3. 3. Which AI use should usually be classified as high risk?

  4. 4. The NIST AI Risk Management Framework is organised around which four functions?

  5. 5. Why should governance offer a fast track for low-risk uses?

Answer all 5 questions to check.

Practise more