Requirements and acceptance criteria for AI
How to specify AI features when outputs can't be predicted exactly: describing behaviour with examples, quality thresholds instead of exact outputs, failure and fallback behaviour, scope and refusal rules, user experience requirements, and acceptance criteria that can actually be tested.
🎯 By the end of this lesson you will be able to:
- Write AI requirements as behaviour, examples and measurable quality thresholds
- Specify what happens when the AI is unsure, wrong or out of scope
- Write acceptance criteria for AI features that QA can test
In this lesson
Why ordinary requirements don't work
A normal requirement says "when the user clicks Save, the record is stored". It either happens or it doesn't. An AI requirement like "summarise the customer's complaint" can be met in thousands of ways, some good and some bad. You can't list the exact expected output, so you specify behaviour, examples and quality levels instead.
The five parts of an AI requirement
- Task and users: what the AI does, for whom, and in which workflow step.
- Examples: a set of real inputs with good outputs, plus examples of unacceptable outputs and why.
- Quality thresholds: measurable targets on an evaluation set, by slice where it matters.
- Failure behaviour: what happens when the AI is unsure, finds nothing, or a service is down.
- Boundaries: what's out of scope, what it must refuse, data it must never show, actions it must never take alone.
Examples are the specification
Examples communicate more than paragraphs of description. A good example set covers typical cases, tricky cases and cases that should be refused or escalated. It becomes the start of the evaluation set (Lesson 20), so BA and QA should build it together with domain experts.
A requirement with examples
Task: summarise a customer complaint for the support agent, in 3 bullet points max, before they open the ticket Good: "• Order #4471 arrived damaged (broken lid) • Customer wants a replacement, not a refund • Second complaint this month" Bad: adds details not in the email; says "refund" when the customer asked for replacement; more than 3 bullets; includes the customer's card number Escalate: threats, legal language or mentions of injury → flag "Priority: review now" instead of summarising
Developers, testers and the business all see the same picture of "good".
Quality thresholds instead of exact outputs
| Vague | Testable |
|---|---|
| Summaries should be accurate | ≥ 95% of summaries contain no facts that are not in the email, on the 200-case evaluation set |
| Should handle all languages | Spanish and French cases score within 5 points of English |
| Should be fast | First words appear within 2 seconds for 95% of requests |
| Should not show personal data | Zero card numbers or passwords in outputs across all evaluation and red-team cases |
| Agents should like it | ≥ 70% of summaries rated useful by agents in the pilot |
Specify the unhappy paths
- Low confidence or nothing found: say so and route to a person; never guess.
- Out of scope: politely decline and point to the right channel.
- AI service unavailable: the workflow still works without AI (the agent reads the email directly).
- User disagrees: easy to edit, reject or report the output; feedback is captured.
- Harmful or sensitive input: escalate according to policy.
User experience requirements
- Make it clear the content is AI-generated and may contain mistakes.
- Show sources or the evidence behind answers where possible.
- Keep people in control: easy to edit, accept or reject.
- Don't make users wait without feedback: stream or show progress.
- Capture feedback in a way that is quick for users and useful for the team.
Acceptance criteria that can be tested
Acceptance criteria for the complaint summary
Given the 200-case evaluation set, when summaries are generated: - ≥ 95% have no unsupported facts (LLM judge calibrated to 90% human agreement) - ≥ 90% have 3 bullets or fewer - 100% of escalation cases are flagged "Priority: review now" - 0 outputs contain card numbers or passwords (automated check) Given the AI service is unavailable, the ticket opens normally without a summary Given the agent clicks "Not helpful", the feedback is stored with the ticket ID
Each line is measurable, and the set covers quality, safety, failure and feedback.
What this means for your role
Developer
Build the failure paths and zero-tolerance controls in code, and ask for examples whenever a requirement is ambiguous.
Tester / QA
Turn the examples and thresholds into the evaluation set and automated checks; challenge any requirement you can't measure.
Business Analyst / PO
Write AI requirements with the five parts, build the example set with domain experts, and specify unhappy paths explicitly.
Tech Lead / Architect
Check that thresholds are achievable and measurable, and that zero-tolerance rules are enforced outside the model.
Project / Delivery Manager
Agree quality thresholds with the business owner before build; they define "done" and the go/no-go gates.
Scrum Master / Agile
Add AI-specific items to the definition of done: evaluation passed, failure paths tested, feedback captured.
Try it: rewrite a vague AI requirement
- Take a vague AI requirement such as "the chatbot should answer HR questions accurately".
- Add three good and two bad example answers, two measurable thresholds and one zero-tolerance rule.
- Write the failure behaviour for "no relevant policy found" and "service unavailable", then three testable acceptance criteria.
Key takeaways
- AI outputs can't be specified exactly, so specify task, examples, thresholds, failure behaviour and boundaries.
- Examples are the clearest specification and seed the evaluation set.
- Turn vague wishes into measurable thresholds, and mark zero-tolerance rules that need hard controls.
- Specify unhappy paths and UX, and write acceptance criteria that QA can measure.
Key terms
- Acceptance criteria
- Conditions a feature must meet to be accepted as done.
- Quality threshold
- A measurable target, such as a minimum score on an evaluation set, that defines acceptable quality.
- Zero-tolerance rule
- A requirement that must never be broken, such as never exposing card numbers.
- Fallback
- What the system does when the AI is unavailable or can't give a good answer.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Why can't AI requirements list the exact expected output?
2. Which requirement is testable?
3. The AI finds no relevant policy. What should the requirement say?
4. "Never show card numbers" is a zero-tolerance rule. How should it be enforced?
5. Who should build the example set for an AI requirement?