An evaluation strategy for AI features
How testers become AI evaluation engineers: defining quality attributes, prioritising by risk, agreeing acceptance thresholds, evaluating at every stage from prototype to production, and reading results with healthy statistical caution.
šÆ By the end of this lesson you will be able to:
- Define the quality attributes that matter for an AI feature and turn them into measurable criteria
- Prioritise evaluation effort by risk and agree acceptance thresholds with stakeholders
- Plan evaluation across stages and judge how much a score can be trusted
In this lesson
From tester to evaluation engineer
Everything that makes a good tester valuable carries over: curiosity, thinking about what can go wrong, building test cases, and reporting clearly. What changes is the nature of the thing under test. An AI feature has no single correct output, gives different outputs for the same input, and has a long tail of rare inputs where it may fail badly. An evaluation engineer's job is to turn "is it good?" into evidence that a team can trust when deciding to ship.
| Traditional testing | AI evaluation |
|---|---|
| Expected output is known | Many acceptable outputs; quality judged by criteria |
| Deterministic: same input, same result | Variable: results are a distribution |
| Pass or fail per test | Scores and rates across a dataset |
| Bugs are fixed and stay fixed | Quality can drift when models, data or prompts change |
| Coverage by requirements | Coverage by requirements, user groups, risks and edge cases |
Define what quality means
Start by listing the quality attributes for this feature, then define each in a way two people would judge the same way. Not every attribute matters equally; agree which ones are critical.
| Attribute | Question | Example criterion |
|---|---|---|
| Correctness | Is it factually right? | Matches the policy or reference answer |
| Groundedness | Is it supported by the sources? | Every claim traceable to a cited passage |
| Completeness | Does it answer every part? | All sub-questions addressed |
| Safety and privacy | Does it avoid harm and leaks? | No personal data of other users; follows content policy |
| Format and tone | Is it usable and appropriate? | Valid JSON; polite, plain language |
| Behaviour when unsure | Does it admit limits? | Says it can't find the answer instead of guessing |
| Cost and speed | Is it affordable and fast? | Under target cost and latency |
Prioritise by risk
You can't test everything equally. Rate each type of failure by likelihood and impact, and spend most effort where both are high. A wrong product recommendation is annoying; a wrong medication dose, a leaked salary or an unauthorised refund is serious.
- Zero-tolerance failures: data leaks, unsafe actions, harmful content. Any single occurrence blocks release.
- Critical quality: core correctness on the most common tasks, measured against a high threshold.
- Important quality: completeness, tone and format, with sensible thresholds.
- Nice to have: style preferences, tracked but not blocking.
Agree acceptance thresholds up front
Before building, agree with the product owner what "good enough to ship" means, written as measurable thresholds per attribute and per user group. This turns release decisions from opinions into evidence, and protects the team from moving goalposts.
Acceptance criteria for a policy assistant
⢠Correctness: at least 92% of answers score 4+ on the rubric, overall and for every region ⢠Groundedness: at least 95% of claims supported by a cited source ⢠No-answer cases: correct refusal in at least 90% ⢠Safety: zero answers revealing another employee's data in 200 adversarial tests ⢠Latency: 95% of answers start within 2 seconds
Each line is testable, has an owner and a number, and includes a per-group check so averages can't hide a weak region.
Evaluate at every stage
| Stage | Question | Methods |
|---|---|---|
| Prototype | Is this approach promising? | Small hand-checked set, quick comparisons of options |
| Development | Is each change an improvement? | Evaluation set run on every prompt, model or retrieval change |
| Pre-release | Does it meet the agreed thresholds? | Full evaluation, red teaming, human review of samples |
| Production | Is it working for real users? | Feedback, sampled reviews, online metrics, A/B tests, drift checks |
How much can you trust a score?
A score from a small set is uncertain. With 100 test cases and a 90% pass rate, the true rate could reasonably be anywhere from about 84% to 96%. With 20 cases, the range is far wider. Two practical habits help:
- Don't over-read small differences. 88% versus 90% on 50 cases is probably noise; look for consistent improvements across runs and slices.
- Run non-deterministic features more than once and report the average and spread, or fix randomness where possible, such as temperature 0 for evaluation runs of structured tasks.
- Grow the set where decisions are close: more cases in the categories that matter most narrow the uncertainty where you need it.
What this means for your role
Developer
Make evaluation easy to run: a single command or pipeline step, with results stored per version so you can compare changes quickly.
Tester / QA
Own the evaluation strategy document: quality attributes, risk ratings, thresholds, datasets, methods per stage and reporting format. This is the core artifact of an AI Evaluation Engineer.
Business Analyst / PO
Work with the evaluation engineer to turn business expectations into measurable criteria and examples of good and bad outputs.
Tech Lead / Architect
Ensure the architecture supports evaluation: versioned prompts and models, logging, the ability to replay inputs and run offline evaluations.
Project / Delivery Manager
Agree acceptance thresholds before development starts, and use evaluation reports, not demos, for go/no-go decisions.
Scrum Master / Agile
Treat evaluation work as first-class backlog items with clear outcomes, not invisible "testing at the end".
Try it: write a one-page evaluation strategy
- Pick an AI feature you know. List five quality attributes and define each in one sentence.
- Rate the main failure types by likelihood and impact, and mark any zero-tolerance ones.
- Write three acceptance thresholds with numbers, including one per-group check.
Key takeaways
- AI evaluation replaces exact expected outputs with criteria, scores and rates across datasets.
- Define quality attributes clearly, prioritise by risk and mark zero-tolerance failures.
- Agree measurable acceptance thresholds, including per-group checks, before building.
- Evaluate at every stage and treat scores with statistical caution: sample size and variability matter.
Key terms
- Quality attribute
- A dimension of quality, such as correctness, groundedness or safety, defined so it can be judged consistently.
- Acceptance threshold
- The minimum measured quality agreed as good enough to release.
- Slice
- A subset of evaluation cases sharing a property, such as a region, language or task type.
- Zero-tolerance failure
- A failure type where a single occurrence blocks release.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. What most distinguishes AI evaluation from traditional testing?
2. Which failure should usually be zero-tolerance?
3. Why include per-group thresholds, such as "for every region"?
4. On 50 cases, version A scores 88% and version B 90%. What is the sensible conclusion?
5. When should acceptance thresholds be agreed?