Evaluation in CI and production
Automating evaluation so every change is checked: fast smoke evals and full runs, comparing against a baseline with tolerances, release gates, handling non-determinism and cost, then online evaluation, A/B tests, drift detection and quality reporting.
๐ฏ By the end of this lesson you will be able to:
- Design an automated evaluation pipeline with smoke and full runs and baseline comparison
- Set release gates that handle non-determinism without becoming flaky
- Continue evaluation in production with online metrics, A/B tests and drift detection
In this lesson
Why evaluation must be automated
An evaluation set run by hand once before launch protects you once. AI features change constantly: prompt tweaks, new documents, model upgrades, library updates. Automating evaluation in your continuous integration (CI) pipeline means every change is measured the same way, regressions are caught before users see them, and the team can move fast with confidence.
The evaluation pipeline
| Run | When | What | Time and cost |
|---|---|---|---|
| Unit tests with mocks | Every commit | Prompt building, parsing, validation, error paths | Seconds, free |
| Smoke eval | Every pull request touching AI behaviour | A small, representative subset (e.g. 30โ50 cases) plus all zero-tolerance tests | Minutes, low |
| Full eval | Before release and nightly | Entire dataset, all slices, judge scoring, attack suite | Longer, moderate |
| Human review | Before major releases | Sample of outputs and all borderline judge scores | Hours, people time |
Compare with a baseline, not a fixed number only
Each run is compared with the current production version's scores, the baseline, as well as with the absolute acceptance thresholds (Lesson 19). A change can pass the thresholds and still be a regression worth investigating.
An evaluation report on a pull request
Prompt change: support-reply v7 โ v8 (dataset v12, 260 cases, 3 runs each) Correctness โฅ4: 91.2% โ 93.1% โ (+1.9) Groundedness: 95.8% โ 95.4% โ within tolerance (โ0.4, tolerance 1.0) No-answer refusals: 90.0% โ 84.0% โ regression beyond tolerance Slice: Spanish 88.0% โ 86.5% โ below threshold 88% Zero-tolerance (leaks, unsafe actions): 0 / 140 โ Cost per request: $0.011 โ $0.012 โ Verdict: BLOCKED โ refusals and Spanish slice need attention
Decision-makers see improvements and regressions at a glance, per slice, with a clear verdict.
Release gates that aren't flaky
- Zero-tolerance tests are hard gates: any failure blocks the release.
- Quality scores use tolerances: block only if a score drops more than an agreed margin, or falls below the threshold.
- Reduce randomness where possible: temperature 0 for structured tasks, pinned model versions, fixed judge prompts.
- Repeat where variable: run open-ended cases several times and compare averages, so one unlucky sample doesn't fail the build.
- Make overrides explicit: a human can approve a known, accepted regression, but the decision and reason are recorded.
Evaluation in production
| Method | What it tells you |
|---|---|
| User feedback and implicit signals | Thumbs, edits, retries, escalations: how real users experience quality |
| Sampled judge and human review | Ongoing quality scores on real traffic, not just the test set |
| A/B tests | Whether a new version actually improves business outcomes, such as resolution rate |
| Guardrail metrics | Safety blocks, leak detections, refusal rates, cost and latency |
| Drift detection | Changes in what users ask, or in score trends, that suggest the dataset or system needs updating |
Close the loop: production failures become new dataset cases, the dataset gets a new version, the baseline is re-run, and the next change is measured against reality, not last year's assumptions.
Reporting quality to stakeholders
- A one-page quality dashboard per AI feature: key scores against thresholds, per-slice status, zero-tolerance results, trend over releases, cost and latency.
- Plain-language release notes about quality: what improved, what got worse and why it is acceptable.
- A clear list of known limitations, so support and product teams can set expectations.
What this means for your role
Developer
Wire evaluation into CI: a runner that loads the versioned dataset, runs the system, scores with checks and judges, compares with the baseline and posts a report on the pull request.
Tester / QA
Define the smoke subset, tolerances, gates and override process, and own the quality dashboard. You are the guardian of the release gate.
Business Analyst / PO
Translate quality reports into business impact for stakeholders, and keep the list of known limitations current.
Tech Lead / Architect
Make evaluation infrastructure shared across AI features: one runner, one report format, cached results and controlled evaluation spend.
Project / Delivery Manager
Use the evaluation report and A/B results for go/no-go and roadmap decisions, and require explicit, recorded approval for accepted regressions.
Scrum Master / Agile
Add the evaluation gate to the definition of done, and review quality trends in sprint reviews alongside features delivered.
Try it: design your evaluation gate
- Choose 30 cases from your dataset as a smoke set, covering every slice and all zero-tolerance tests.
- Write the gate rules: hard fails, tolerances per score and the minimum thresholds.
- Draft the pull-request report format, using the example above, and decide who can approve an override.
Key takeaways
- Automate evaluation in CI so every AI change is measured consistently.
- Use layered runs: mocks on every commit, a smoke eval on pull requests, full runs before release and nightly.
- Gate on zero-tolerance failures and on regressions beyond agreed tolerances, managing randomness and cost.
- Keep evaluating in production with feedback, sampled review, A/B tests and drift detection, and feed failures back into the dataset.
Key terms
- Continuous integration (CI)
- Automatically building and testing every change before it is merged.
- Baseline
- The scores of the current production version that new changes are compared with.
- Tolerance
- The allowed drop in a score before a change is treated as a regression.
- Drift
- A gradual change in inputs or behaviour that makes past evaluation less representative.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. What should run on every pull request that changes AI behaviour?
2. A change keeps overall correctness above the threshold but drops refusals well beyond tolerance. What should happen?
3. How do you stop non-determinism from making the gate flaky?
4. Which shows whether a new version improves real business outcomes?
5. What should happen to production failures?