Data requirements and readiness
What data an AI feature needs and whether it is ready: the four kinds of data (knowledge, context, examples, feedback), data quality dimensions, ownership and freshness, privacy and permission limits, and a readiness checklist to use before committing to build.
๐ฏ By the end of this lesson you will be able to:
- Identify the four kinds of data an AI feature needs
- Assess data quality, ownership, freshness and access
- Respect privacy and permission limits, and judge data readiness before building
In this lesson
AI is only as good as its data
Most AI features in business don't train their own model, but they still depend heavily on data. A policy assistant is only as good as the policy documents it searches. A complaint summary is only as useful as the ticket data it can see. Data problems discovered late are one of the most common reasons AI projects stall (Lesson 28).
The four kinds of data
| Kind | What it is | Example (support assistant) |
|---|---|---|
| Knowledge | Documents and records the AI retrieves answers from | Policies, product manuals, help articles |
| Context | Information about the current user or case | The ticket, order history, customer tier |
| Examples | Inputs with known good outputs, for evaluation and few-shot prompts | 200 past tickets with expert-approved replies |
| Feedback | Signals from real use | Accepted, edited or rejected drafts; thumbs up or down |
Data quality dimensions
| Dimension | Question | AI symptom when it's poor |
|---|---|---|
| Accuracy | Is it correct? | Confident answers based on wrong information |
| Completeness | Is anything missing? | "I can't find that" for common questions |
| Freshness | Is it up to date? | Answers based on last year's policy |
| Consistency | Do sources agree? | Different answers to the same question |
| Structure | Is it readable by machines? | Scanned PDFs, tables broken in extraction |
| Coverage | Does it cover all users, languages and cases? | Poor results for some regions or languages |
Ownership, freshness and access
- Owner: who is responsible for each source being correct? Without owners, content goes stale.
- Update process: when a policy changes, how quickly does the AI see it? Who triggers re-indexing?
- Access: can the system technically reach the data (APIs, exports), and is it allowed to?
- Single source of truth: remove or mark duplicates and old versions so retrieval doesn't find them.
Privacy and permission limits
- Purpose: is using this data for this AI feature compatible with why it was collected? Check with privacy colleagues.
- Minimisation: include only the fields the task needs; mask the rest.
- Permissions: users must only see AI answers based on data they are allowed to see.
- Sensitive categories: health, financial, children's and similar data need extra care and approval.
- Evaluation data: real examples may contain personal data; anonymise them or control access to the evaluation set.
- Retention: how long are prompts, outputs and feedback kept?
A data readiness checklist
Before committing to build
โ Each needed source identified, with an owner โ Access is technically possible and approved โ Quality checked on a sample: accurate, complete, current, consistent โ Outdated and duplicate content removed or marked โ Update process defined (who, how often) โ Permission rules defined for who can see what โ Privacy review done: purpose, minimisation, sensitive data, retention โ At least 100 realistic examples collected for evaluation โ Plan to capture feedback from real use
Unticked boxes are risks or work items for the plan, not reasons to ignore the gap.
What this means for your role
Developer
Build ingestion that respects permissions, handles messy formats, tracks the source version and re-indexes when content changes.
Tester / QA
Test with stale, conflicting and restricted documents to make sure the AI picks current sources and never leaks restricted ones.
Business Analyst / PO
Own the data requirements: list sources, owners, quality issues, permission rules and the readiness checklist.
Tech Lead / Architect
Design ingestion, permission filtering and freshness into the architecture, and flag inaccessible data early.
Project / Delivery Manager
Plan and fund data clean-up and ownership as real work, and treat the readiness checklist as a gate before build.
Scrum Master / Agile
Make data work visible on the board and help the team get time from content owners.
Try it: run a data readiness check
- Pick an AI use case and list the knowledge, context, examples and feedback data it would need.
- For one knowledge source, check 10 documents for accuracy, freshness and duplicates.
- Go through the readiness checklist and turn each unticked box into a work item with an owner.
Key takeaways
- AI features depend on four kinds of data: knowledge, context, examples and feedback.
- Check accuracy, completeness, freshness, consistency, structure and coverage; data clean-up often beats prompt tweaks.
- Every source needs an owner, an update process and approved access.
- Respect purpose, minimisation, permissions and retention, and use a readiness checklist before building.
Key terms
- Data readiness
- Whether the data a feature needs is available, accessible, good enough and approved for use.
- Data owner
- The person responsible for a data source being accurate, current and used correctly.
- Permission filtering
- Making sure AI answers only use data the current user is allowed to see.
- Anonymisation
- Removing or altering personal details so individuals can't be identified.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. The AI gives answers based on last year's policy. Which data quality dimension failed?
2. Which kind of data are past tickets with expert-approved replies?
3. All company documents are indexed into one search store without permission filters. What is the risk?
4. Three conflicting versions of a policy are in the knowledge base. What usually helps most?
5. What should happen to unticked items on the data readiness checklist?