Nishaglobal Education logo

An evaluation strategy for AI features

How testers become AI evaluation engineers: defining quality attributes, prioritising by risk, agreeing acceptance thresholds, evaluating at every stage from prototype to production, and reading results with healthy statistical caution.

Lesson 19 of 4418 min readQuiz: 5 questions

šŸŽÆ By the end of this lesson you will be able to:

  • Define the quality attributes that matter for an AI feature and turn them into measurable criteria
  • Prioritise evaluation effort by risk and agree acceptance thresholds with stakeholders
  • Plan evaluation across stages and judge how much a score can be trusted
In this lesson
  1. From tester to evaluation engineer
  2. Define what quality means
  3. Prioritise by risk
  4. Agree acceptance thresholds up front
  5. Evaluate at every stage
  6. How much can you trust a score?
  7. What this means for your role
  8. Key takeaways
  9. Quiz

From tester to evaluation engineer

Everything that makes a good tester valuable carries over: curiosity, thinking about what can go wrong, building test cases, and reporting clearly. What changes is the nature of the thing under test. An AI feature has no single correct output, gives different outputs for the same input, and has a long tail of rare inputs where it may fail badly. An evaluation engineer's job is to turn "is it good?" into evidence that a team can trust when deciding to ship.

Traditional testingAI evaluation
Expected output is knownMany acceptable outputs; quality judged by criteria
Deterministic: same input, same resultVariable: results are a distribution
Pass or fail per testScores and rates across a dataset
Bugs are fixed and stay fixedQuality can drift when models, data or prompts change
Coverage by requirementsCoverage by requirements, user groups, risks and edge cases

Define what quality means

Start by listing the quality attributes for this feature, then define each in a way two people would judge the same way. Not every attribute matters equally; agree which ones are critical.

AttributeQuestionExample criterion
CorrectnessIs it factually right?Matches the policy or reference answer
GroundednessIs it supported by the sources?Every claim traceable to a cited passage
CompletenessDoes it answer every part?All sub-questions addressed
Safety and privacyDoes it avoid harm and leaks?No personal data of other users; follows content policy
Format and toneIs it usable and appropriate?Valid JSON; polite, plain language
Behaviour when unsureDoes it admit limits?Says it can't find the answer instead of guessing
Cost and speedIs it affordable and fast?Under target cost and latency

Prioritise by risk

You can't test everything equally. Rate each type of failure by likelihood and impact, and spend most effort where both are high. A wrong product recommendation is annoying; a wrong medication dose, a leaked salary or an unauthorised refund is serious.

  • Zero-tolerance failures: data leaks, unsafe actions, harmful content. Any single occurrence blocks release.
  • Critical quality: core correctness on the most common tasks, measured against a high threshold.
  • Important quality: completeness, tone and format, with sensible thresholds.
  • Nice to have: style preferences, tracked but not blocking.

Agree acceptance thresholds up front

Before building, agree with the product owner what "good enough to ship" means, written as measurable thresholds per attribute and per user group. This turns release decisions from opinions into evidence, and protects the team from moving goalposts.

Acceptance criteria for a policy assistant

• Correctness: at least 92% of answers score 4+ on the rubric, overall and for every region
• Groundedness: at least 95% of claims supported by a cited source
• No-answer cases: correct refusal in at least 90%
• Safety: zero answers revealing another employee's data in 200 adversarial tests
• Latency: 95% of answers start within 2 seconds

Each line is testable, has an owner and a number, and includes a per-group check so averages can't hide a weak region.

Evaluate at every stage

Prototypepromising?small hand-checked setDevelopmentbetter each change?eval set per changePre-releasemeets thresholds?full eval + red teamProductionworking for users?feedback + A/B + driftEach stage answers a different question
Different evaluations answer different questions as a feature moves from prototype to production.
StageQuestionMethods
PrototypeIs this approach promising?Small hand-checked set, quick comparisons of options
DevelopmentIs each change an improvement?Evaluation set run on every prompt, model or retrieval change
Pre-releaseDoes it meet the agreed thresholds?Full evaluation, red teaming, human review of samples
ProductionIs it working for real users?Feedback, sampled reviews, online metrics, A/B tests, drift checks

How much can you trust a score?

A score from a small set is uncertain. With 100 test cases and a 90% pass rate, the true rate could reasonably be anywhere from about 84% to 96%. With 20 cases, the range is far wider. Two practical habits help:

  • Don't over-read small differences. 88% versus 90% on 50 cases is probably noise; look for consistent improvements across runs and slices.
  • Run non-deterministic features more than once and report the average and spread, or fix randomness where possible, such as temperature 0 for evaluation runs of structured tasks.
  • Grow the set where decisions are close: more cases in the categories that matter most narrow the uncertainty where you need it.

What this means for your role

Developer

Make evaluation easy to run: a single command or pipeline step, with results stored per version so you can compare changes quickly.

Tester / QA

Own the evaluation strategy document: quality attributes, risk ratings, thresholds, datasets, methods per stage and reporting format. This is the core artifact of an AI Evaluation Engineer.

Business Analyst / PO

Work with the evaluation engineer to turn business expectations into measurable criteria and examples of good and bad outputs.

Tech Lead / Architect

Ensure the architecture supports evaluation: versioned prompts and models, logging, the ability to replay inputs and run offline evaluations.

Project / Delivery Manager

Agree acceptance thresholds before development starts, and use evaluation reports, not demos, for go/no-go decisions.

Scrum Master / Agile

Treat evaluation work as first-class backlog items with clear outcomes, not invisible "testing at the end".

Try it: write a one-page evaluation strategy

  1. Pick an AI feature you know. List five quality attributes and define each in one sentence.
  2. Rate the main failure types by likelihood and impact, and mark any zero-tolerance ones.
  3. Write three acceptance thresholds with numbers, including one per-group check.

Key takeaways

  • AI evaluation replaces exact expected outputs with criteria, scores and rates across datasets.
  • Define quality attributes clearly, prioritise by risk and mark zero-tolerance failures.
  • Agree measurable acceptance thresholds, including per-group checks, before building.
  • Evaluate at every stage and treat scores with statistical caution: sample size and variability matter.

Key terms

Quality attribute
A dimension of quality, such as correctness, groundedness or safety, defined so it can be judged consistently.
Acceptance threshold
The minimum measured quality agreed as good enough to release.
Slice
A subset of evaluation cases sharing a property, such as a region, language or task type.
Zero-tolerance failure
A failure type where a single occurrence blocks release.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. What most distinguishes AI evaluation from traditional testing?

  2. 2. Which failure should usually be zero-tolerance?

  3. 3. Why include per-group thresholds, such as "for every region"?

  4. 4. On 50 cases, version A scores 88% and version B 90%. What is the sensible conclusion?

  5. 5. When should acceptance thresholds be agreed?

Answer all 5 questions to check.

Practise more