Evaluating AI quality
Why pass/fail tests are not enough for AI, how to build an evaluation set, which measures to use (from exact match to rubrics, human review and AI judges), and how evaluation becomes a habit, not a one-off.
šÆ By the end of this lesson you will be able to:
- Explain why AI features need evaluation sets instead of single test cases
- Choose suitable measures: exact match, precision and recall, rubric scores, human review and LLM-as-judge
- Set up a simple evaluation cycle that catches regressions before users do
In this lesson
Why AI needs a different kind of testing
Traditional tests check that a given input produces one exact output. AI features break this in two ways: the same input can give different outputs, and there are many acceptable answers to most questions. A summary can be good in a hundred different wordings. So instead of asking "is this output exactly right?", we ask "across many realistic examples, how often is the output good enough, and how bad are the failures?" That is evaluation, often shortened to evals.
Building an evaluation set
An evaluation set is a collection of realistic inputs, each with the expected answer or the criteria a good answer must meet. Start small, around 30 to 50 cases, and grow it whenever you find a new failure.
- Typical cases: the everyday questions and inputs most users send.
- Edge cases: very short or long inputs, typos, other languages, unusual formats.
- Hard and tricky cases: ambiguous questions, misleading context, questions with no answer in the sources.
- Safety cases: requests for personal data, prompt injection attempts, harmful requests.
- Coverage by group: examples from different user types, regions and languages, to spot unfair gaps.
Choosing how to measure
| Measure | When to use it | Example |
|---|---|---|
| Exact match or valid structure | One right answer, or a required format | Category is "billing"; JSON has all fields |
| Accuracy, precision and recall | Classification and detection tasks | Fraud flags, ticket routing, spam detection |
| Rubric scores | Open-ended answers | Score 1 to 5 on correctness, completeness, tone |
| Human review | High-stakes or subtle quality | Expert checks a sample of answers each week |
| LLM-as-judge | Scoring many open answers quickly | A model grades answers against the rubric |
| Faithfulness | RAG and summaries | Is every claim supported by the source? |
A simple rubric for a support answer
Score each answer from 1 (poor) to 5 (excellent) on: ā¢ Correct: matches policy and facts ā¢ Complete: answers every part of the question ā¢ Grounded: no claims beyond the sources ā¢ Clear and kind: plain language, appropriate tone Fail automatically if it reveals personal data or promises something not in policy.
A rubric turns "good answer" into something people, or an AI judge, can score consistently.
Evaluation as a cycle
- Baseline: run the eval set on the current version and record the scores.
- Change: adjust a prompt, model, setting or document.
- Compare: run the same set again. Did scores improve? Did any category get worse (a regression)?
- Investigate and learn: read the failures, fix the cause, and add new cases for anything you discover.
- Monitor in production: track user feedback, escalations and sampled reviews. Real use always reveals cases you didn't think of.
What this means for your role
Developer
Automate the eval run so it can be triggered on every prompt or model change, and store results to compare versions. Block releases that regress on key cases.
Tester / QA
This is the heart of the AI Evaluation Engineer role: designing eval sets, rubrics and judges, measuring by category, and reporting quality clearly. Your testing instincts transfer directly.
Business Analyst / PO
Write acceptance criteria as quality thresholds, such as "at least 90% of answers score 4 or higher on correctness, and zero data leaks", plus the examples that define good.
Tech Lead / Architect
Make evaluation part of the architecture: logging inputs and outputs, version tags, and pipelines that run evals before and after release.
Project / Delivery Manager
Agree target scores and "must never happen" failures before building. Use eval results, not demos, to make go and no-go decisions.
Scrum Master / Agile
Add "eval set updated and passing" to the definition of done for AI work, and review new failure types in retrospectives.
Try it: score an AI assistant
- Write ten questions you know the answers to, including two with no correct answer available.
- Ask an AI assistant each question and score the answers with the rubric above.
- Change one thing, such as adding "If you're not sure, say so", run all ten again and compare the scores.
Key takeaways
- AI outputs vary and have many acceptable forms, so quality is measured across an evaluation set, not single tests.
- Good eval sets include typical, edge, tricky, safety and group-coverage cases, and grow with every new failure.
- Pick measures to fit the task: exact match, precision and recall, rubrics, human review, LLM-as-judge and faithfulness.
- Evaluate on every change, watch for regressions and keep monitoring after release.
Key terms
- Evaluation set
- A collection of realistic inputs with expected answers or quality criteria, used to measure an AI feature.
- Rubric
- A scoring guide that breaks quality into named criteria with levels.
- Precision / recall
- Precision: share of flagged items that were right. Recall: share of real cases that were caught.
- LLM-as-judge
- Using a language model to score outputs against a rubric.
- Regression
- A change that makes previously good behaviour worse.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Why aren't exact-output tests enough for most AI features?
2. A filter flags 50 items; 40 are real problems. There were 100 real problems in total. What is its recall?
3. Which belongs in an evaluation set for a RAG assistant?
4. Before relying on an LLM-as-judge, you should:
5. After changing a prompt, accuracy rises overall but drops for Spanish questions. This is: