Metrics, rubrics and LLM-as-judge
Choosing the right kind of check for each quality attribute, from automatic checks to rubrics, designing rubrics that people and models score consistently, pairwise comparisons, and building and calibrating an AI judge you can trust.
๐ฏ By the end of this lesson you will be able to:
- Match each quality attribute to the cheapest reliable check: code, reference, rubric or human
- Design rubrics with clear criteria, levels and examples
- Build an LLM judge, detect its biases and calibrate it against human scores
In this lesson
Use the cheapest check that is reliable
Not every quality question needs a human or another AI. Climb the ladder only as far as needed: automatic checks are cheap, fast and consistent; human review is expensive but catches what nothing else can.
| Check type | Good for | Example |
|---|---|---|
| Code checks | Format, structure, rules | Valid JSON, required fields present, no email addresses, under 120 words |
| Reference match | One right answer | Category equals "billing"; extracted date equals 2026-03-31 |
| Must-include / must-not-include | Key facts in free text | Mentions "5 days"; doesn't promise a refund |
| Rubric scored by an LLM judge | Open-ended quality at scale | Correctness and completeness scores 1โ5 |
| Human review | Subtle, high-stakes or new behaviour | Expert checks 50 sampled answers per release |
Designing rubrics
- One criterion at a time: score correctness, completeness and tone separately rather than one overall "quality" number.
- Observable descriptions: "states the correct deadline from the source" beats "is accurate".
- Few levels: 1โ3 or pass/fail is easier to apply consistently than 1โ10.
- Anchor examples: show a real example of each level.
- Clear automatic fails: for example, any invented policy detail or any personal data from another user.
A three-level correctness rubric
3 = All facts match the source; nothing important missing 2 = Main answer correct; a minor detail missing or imprecise 1 = Any wrong fact, invented detail or contradiction of the source Automatic 1: states a number, date or rule not in the source
Short, observable and easy to apply. The "automatic 1" rule catches hallucinations decisively.
Absolute scores or pairwise comparison?
When comparing two versions, such as two prompts, it is often more reliable to show a reviewer or judge both answers to the same input and ask which is better on a given criterion than to score each separately. People and models are better at comparing than at assigning consistent absolute numbers. Use absolute scores for release thresholds, and pairwise comparisons for choosing between versions.
Building an LLM judge
- Give the judge the input, the output, the reference or source, and one rubric criterion.
- Ask it to explain its reasoning briefly and then give the score, in structured output.
- Use a capable model, low temperature, and a fixed, versioned judge prompt.
- Never let the judge see which version produced which answer.
| Known judge bias | What it looks like | Mitigation |
|---|---|---|
| Position bias | Prefers the first (or second) answer in a comparison | Run both orders; count only consistent wins |
| Verbosity bias | Rates longer answers higher | Rubric rewards concision; check length correlation |
| Self-preference | Favours answers in its own style or from its own model family | Use a different model as judge, or validate against humans |
| Leniency | Gives high scores too easily | Anchor examples; strict automatic fails |
Calibrate the judge against humans
Take 50 to 100 outputs, have humans score them with the rubric, and have the judge score the same outputs. Measure how often they agree, and look at the disagreements. Improve the judge prompt or rubric until agreement is high for each criterion. Agreement measures that account for chance, such as Cohen's kappa, give a more honest picture than simple percentage agreement. Re-check every time you change the judge, the rubric or the model being judged.
What this means for your role
Developer
Implement checks as a library: code checks, reference comparisons and a judge call with structured output, all versioned and reusable across features.
Tester / QA
Own the rubrics and the judge's calibration: agreement measurements, bias checks and the human review sample. Report which criteria are judge-scored and how well they agree with people.
Business Analyst / PO
Help write observable rubric descriptions and anchor examples from the business point of view.
Tech Lead / Architect
Decide which judge models are approved, keep judge prompts under change control, and budget evaluation compute.
Project / Delivery Manager
Ask how each reported metric is measured, and whether AI-judged scores have been checked against human reviewers.
Scrum Master / Agile
Schedule recurring calibration sessions where team members score the same samples, which keeps the team's idea of quality aligned.
Try it: test a judge
- Write a three-level rubric for one criterion of an AI feature you know.
- Score ten answers yourself, then ask an AI model to score the same ten using your rubric, explaining before scoring.
- Count agreements, read every disagreement, and adjust the rubric wording. Then swap the order of two answers in a comparison and see if the judge changes its mind.
Key takeaways
- Use the cheapest reliable check: code, reference match, must-include points, judge rubrics, then humans.
- Good rubrics are per-criterion, observable, few-level and anchored with examples, with automatic fails.
- Pairwise comparisons are better for choosing between versions; absolute scores for release thresholds.
- LLM judges have biases; calibrate them against human scores and keep re-checking.
Key terms
- Rubric
- A scoring guide with criteria, levels and examples.
- Pairwise comparison
- Judging which of two outputs is better for the same input.
- Position bias
- A judge's tendency to prefer an answer because of where it appears.
- Calibration
- Checking and adjusting a judge until its scores agree well with expert human scores.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. How should you check that an output is valid JSON with required fields?
2. Which rubric description is best?
3. A judge prefers whichever answer it sees first. How do you reduce this?
4. Before relying on an LLM judge for a criterion, you should:
5. When choosing between two prompt versions, which method is often more reliable?