Nishaglobal Education logo

Metrics, rubrics and LLM-as-judge

Choosing the right kind of check for each quality attribute, from automatic checks to rubrics, designing rubrics that people and models score consistently, pairwise comparisons, and building and calibrating an AI judge you can trust.

Lesson 21 of 4419 min readQuiz: 5 questions

๐ŸŽฏ By the end of this lesson you will be able to:

  • Match each quality attribute to the cheapest reliable check: code, reference, rubric or human
  • Design rubrics with clear criteria, levels and examples
  • Build an LLM judge, detect its biases and calibrate it against human scores
In this lesson
  1. Use the cheapest check that is reliable
  2. Designing rubrics
  3. Absolute scores or pairwise comparison?
  4. Building an LLM judge
  5. Calibrate the judge against humans
  6. What this means for your role
  7. Key takeaways
  8. Quiz

Use the cheapest check that is reliable

Not every quality question needs a human or another AI. Climb the ladder only as far as needed: automatic checks are cheap, fast and consistent; human review is expensive but catches what nothing else can.

Check typeGood forExample
Code checksFormat, structure, rulesValid JSON, required fields present, no email addresses, under 120 words
Reference matchOne right answerCategory equals "billing"; extracted date equals 2026-03-31
Must-include / must-not-includeKey facts in free textMentions "5 days"; doesn't promise a refund
Rubric scored by an LLM judgeOpen-ended quality at scaleCorrectness and completeness scores 1โ€“5
Human reviewSubtle, high-stakes or new behaviourExpert checks 50 sampled answers per release

Designing rubrics

  • One criterion at a time: score correctness, completeness and tone separately rather than one overall "quality" number.
  • Observable descriptions: "states the correct deadline from the source" beats "is accurate".
  • Few levels: 1โ€“3 or pass/fail is easier to apply consistently than 1โ€“10.
  • Anchor examples: show a real example of each level.
  • Clear automatic fails: for example, any invented policy detail or any personal data from another user.

A three-level correctness rubric

3 = All facts match the source; nothing important missing
2 = Main answer correct; a minor detail missing or imprecise
1 = Any wrong fact, invented detail or contradiction of the source
Automatic 1: states a number, date or rule not in the source

Short, observable and easy to apply. The "automatic 1" rule catches hallucinations decisively.

Absolute scores or pairwise comparison?

When comparing two versions, such as two prompts, it is often more reliable to show a reviewer or judge both answers to the same input and ask which is better on a given criterion than to score each separately. People and models are better at comparing than at assigning consistent absolute numbers. Use absolute scores for release thresholds, and pairwise comparisons for choosing between versions.

Building an LLM judge

  1. Give the judge the input, the output, the reference or source, and one rubric criterion.
  2. Ask it to explain its reasoning briefly and then give the score, in structured output.
  3. Use a capable model, low temperature, and a fixed, versioned judge prompt.
  4. Never let the judge see which version produced which answer.
Known judge biasWhat it looks likeMitigation
Position biasPrefers the first (or second) answer in a comparisonRun both orders; count only consistent wins
Verbosity biasRates longer answers higherRubric rewards concision; check length correlation
Self-preferenceFavours answers in its own style or from its own model familyUse a different model as judge, or validate against humans
LeniencyGives high scores too easilyAnchor examples; strict automatic fails

Calibrate the judge against humans

100 outputssame sampleHuman scoreswith the rubricJudge scoressame rubricAgreemente.g. 92% ยท kappalow? fix rubric orjudge prompt, re-test
Score the same sample with humans and with the judge. Only rely on the judge for a criterion once agreement is high, and keep re-checking a sample.

Take 50 to 100 outputs, have humans score them with the rubric, and have the judge score the same outputs. Measure how often they agree, and look at the disagreements. Improve the judge prompt or rubric until agreement is high for each criterion. Agreement measures that account for chance, such as Cohen's kappa, give a more honest picture than simple percentage agreement. Re-check every time you change the judge, the rubric or the model being judged.

What this means for your role

Developer

Implement checks as a library: code checks, reference comparisons and a judge call with structured output, all versioned and reusable across features.

Tester / QA

Own the rubrics and the judge's calibration: agreement measurements, bias checks and the human review sample. Report which criteria are judge-scored and how well they agree with people.

Business Analyst / PO

Help write observable rubric descriptions and anchor examples from the business point of view.

Tech Lead / Architect

Decide which judge models are approved, keep judge prompts under change control, and budget evaluation compute.

Project / Delivery Manager

Ask how each reported metric is measured, and whether AI-judged scores have been checked against human reviewers.

Scrum Master / Agile

Schedule recurring calibration sessions where team members score the same samples, which keeps the team's idea of quality aligned.

Try it: test a judge

  1. Write a three-level rubric for one criterion of an AI feature you know.
  2. Score ten answers yourself, then ask an AI model to score the same ten using your rubric, explaining before scoring.
  3. Count agreements, read every disagreement, and adjust the rubric wording. Then swap the order of two answers in a comparison and see if the judge changes its mind.

Key takeaways

  • Use the cheapest reliable check: code, reference match, must-include points, judge rubrics, then humans.
  • Good rubrics are per-criterion, observable, few-level and anchored with examples, with automatic fails.
  • Pairwise comparisons are better for choosing between versions; absolute scores for release thresholds.
  • LLM judges have biases; calibrate them against human scores and keep re-checking.

Key terms

Rubric
A scoring guide with criteria, levels and examples.
Pairwise comparison
Judging which of two outputs is better for the same input.
Position bias
A judge's tendency to prefer an answer because of where it appears.
Calibration
Checking and adjusting a judge until its scores agree well with expert human scores.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. How should you check that an output is valid JSON with required fields?

  2. 2. Which rubric description is best?

  3. 3. A judge prefers whichever answer it sees first. How do you reduce this?

  4. 4. Before relying on an LLM judge for a criterion, you should:

  5. 5. When choosing between two prompt versions, which method is often more reliable?

Answer all 5 questions to check.

Practise more