Nishaglobal Education logo

Building evaluation datasets

Where good test cases come from, how to cover tasks, user groups and risks, writing references and labelling guidelines, using AI to generate cases safely, checking agreement between reviewers, and versioning and refreshing datasets.

Lesson 20 of 4418 min readQuiz: 5 questions

๐ŸŽฏ By the end of this lesson you will be able to:

  • Collect evaluation cases from real usage, experts and careful synthetic generation
  • Design coverage across tasks, difficulty, user groups and risks, tagged as slices
  • Write references and labelling guidelines, measure reviewer agreement, and keep datasets versioned and fresh
In this lesson
  1. The dataset is the real specification
  2. Where cases come from
  3. Design for coverage
  4. References and labelling guidelines
  5. Dataset hygiene
  6. What this means for your role
  7. Key takeaways
  8. Quiz

The dataset is the real specification

Requirements say what an AI feature should do; the evaluation dataset shows it, case by case. Every decision the team makes, such as which prompt, which model or whether to ship, will be judged by this dataset. If it is unrepresentative, every decision based on it is unreliable. Building and caring for evaluation datasets is the most valuable thing an evaluation engineer does.

Where cases come from

SourceStrengthWatch out for
Real usage (anonymised logs, tickets, searches)Shows what users actually askPersonal data must be removed; may miss rare cases
Subject-matter expertsCorrect references, tricky edge casesExpensive; experts may write unnaturally clean inputs
Past incidents and feedbackProven failure modesOnly covers problems already seen
Synthetic generation with AIFast variety: paraphrases, typos, languages, edge casesCan be repetitive or unrealistic; needs human review
Adversarial cases (Lesson 22)Safety and security coverageMust be clearly separated and handled carefully

Design for coverage

EasyTypicalHardAdversarialPolicy Q&A12301510Summarise820106Extract fields1018gap!4No answer / out of scopegap!865Also tag each case by language, region and user group
Plan coverage as a grid of task types against difficulty, then make sure each user group and risk area appears. Empty cells are blind spots.
  • Task types: every distinct job the feature does, such as answering policy questions, summarising, or extracting fields.
  • Difficulty: easy, typical, hard and adversarial.
  • Input variety: short and long, typos, informal language, other languages, attachments, multi-part questions.
  • User groups: regions, languages, roles and accessibility needs, so fairness gaps show up.
  • Special behaviours: questions with no answer, ambiguous requests, out-of-scope requests.

Tag every case with its slices, such as task type, difficulty, language and region. Tags let you report scores per slice, which is where regressions and unfairness become visible.

References and labelling guidelines

Each case needs a way to judge the output. Sometimes that is an exact answer, such as the correct category. More often it is a reference answer plus must-include and must-not-include points, such as "mentions the 31 March deadline" and "does not promise a refund".

One evaluation case

id: policy-carryover-017
input: "can i keep my unused holiday days for next yr?? i have 7 left"
tags: task=policy_qa, difficulty=typical, region=UK, style=informal
reference: "You can carry over up to 5 days, to be used by 31 March."
must_include: ["5 days maximum", "31 March deadline"]
must_not_include: ["all 7 days can be carried over"]
source: Leave Policy v4 ยง3.2

Clear enough that two reviewers, or an AI judge with the rubric, would score an answer the same way.

Write short labelling guidelines: what each score level means, with examples, and how to handle borderline cases. Then have two people score the same sample independently and compare. If they disagree often, the guidelines or rubric need work, because the scores are measuring the reviewers, not the AI.

Dataset hygiene

  • Keep a held-out set: don't use evaluation cases as examples in prompts, or scores will look better than reality.
  • Version the dataset: record what changed and why, so scores from different versions aren't compared blindly.
  • Refresh regularly: add new failure cases from production every week; retire cases that no longer reflect the product.
  • Protect privacy: anonymise real data, restrict access and follow your data policy.
  • Balance size and cost: a focused set of a few hundred well-chosen cases often beats thousands of near-duplicates.

What this means for your role

Developer

Store datasets in version control or a dataset tool with IDs, tags and references, and build the runner to report per-slice scores automatically.

Tester / QA

Run dataset design like test design: coverage matrices, equivalence classes, boundary cases and a living backlog of new cases from production failures.

Business Analyst / PO

Supply realistic inputs and correct references from the business side, and review must-include and must-not-include points for accuracy.

Tech Lead / Architect

Provide a safe pipeline from production logs to anonymised evaluation cases, with access controls and retention rules.

Project / Delivery Manager

Budget expert time for writing references and reviewing cases. Dataset quality limits every quality claim the team can make.

Scrum Master / Agile

Make adding a test case part of every AI bug fix, so the dataset grows with each lesson learned.

Try it: build a 20-case starter set

  1. For an AI feature you know, draw a grid of three task types against easy, typical, hard and adversarial.
  2. Write 20 cases to fill the grid, each with tags, a reference or must-include points, and at least three no-answer or out-of-scope cases.
  3. Ask a colleague to score five outputs with your guidelines and compare with your scores. Improve the guidelines where you disagreed.

Key takeaways

  • The evaluation dataset is the practical specification; every quality decision depends on it.
  • Combine real usage, expert cases, past incidents, reviewed synthetic data and adversarial cases.
  • Design coverage across tasks, difficulty, input variety, user groups and special behaviours, and tag slices.
  • Write references and guidelines, check reviewer agreement, keep a held-out set, and version and refresh the data.

Key terms

Reference answer
An example of a correct answer used to judge outputs.
Labelling guidelines
Instructions that tell reviewers how to score outputs consistently.
Inter-annotator agreement
How often independent reviewers give the same score to the same output.
Held-out set
Evaluation cases kept separate from anything used to build or tune the system.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. Why tag evaluation cases with slices such as region and task type?

  2. 2. What is the main risk of AI-generated test cases?

  3. 3. Two reviewers often give different scores to the same outputs. What does this suggest?

  4. 4. Why keep evaluation cases out of the prompt examples?

  5. 5. A production failure is fixed. What should also happen?

Answer all 5 questions to check.

Practise more