Nishaglobal Education logo

Shipping and monitoring AI features

Releasing AI features safely with flags, shadow runs and gradual rollouts, tracing every request, collecting user feedback, building dashboards and alerts for quality, cost and speed, and handling incidents and model changes.

Lesson 18 of 4417 min readQuiz: 5 questions

🎯 By the end of this lesson you will be able to:

  • Plan a safe release using feature flags, shadow mode and gradual rollout
  • Define what to log and monitor for AI features: quality, cost, latency, errors and safety
  • Respond to AI incidents and manage provider model updates without surprises
In this lesson
  1. Launch is the start, not the finish
  2. Releasing safely
  3. Trace every request
  4. Feedback: your best source of new test cases
  5. Dashboards and alerts
  6. When things go wrong
  7. Managing model changes
  8. What this means for your role
  9. Key takeaways
  10. Quiz

Launch is the start, not the finish

Normal software mostly behaves the same after release as in testing. AI features keep surprising you: real users ask things your evaluation set never imagined, documents change, costs move with usage, and providers update models. Successful teams treat AI features as living systems they watch and improve every week.

Releaseflag, shadow, 1→100%Observetraces + dashboardsLearnfeedback + samplesImprovefix + re-run evalsIncident?flag off → trace → fix
Release gradually, observe real behaviour, learn from feedback and failures, improve, and re-evaluate before the next release.

Releasing safely

TechniqueHow it worksWhy
Feature flagTurn the feature on or off without a deployInstant rollback and targeted access
Shadow modeThe AI runs on real traffic but its output isn't shown; compare with what happenedMeasure quality on real inputs with zero user risk
Internal betaStaff use it firstFast feedback from forgiving users
Gradual rollout1%, then 10%, then 50%, then everyone, with checks between stepsLimits the impact of problems
A/B testCompare against the old experience or another versionProves real-world value, not just demo quality

Trace every request

When a user reports a bad answer, you need to see exactly what happened. Log a trace for each AI request, protecting personal data as Lesson 17 describes:

  • Request ID, user or tenant (pseudonymised where possible), feature and timestamp
  • Model and version, prompt template version and key settings
  • Retrieved documents or chunk IDs, and tool calls with inputs and results
  • Input and output token counts, cost, latency and time to first token
  • Validation results, errors, retries and fallbacks used
  • User feedback attached later, such as thumbs up or down, edits or escalation

Feedback: your best source of new test cases

Make feedback easy: thumbs up or down with an optional comment, "report a problem", and implicit signals such as users editing the AI's draft heavily, retrying or asking for a human. Review negative feedback every week, find the patterns, fix the causes and add the examples to your evaluation set (Lesson 11).

Dashboards and alerts

AreaWatchAlert when
QualityFeedback rate, sampled rubric scores, escalation rateNegative feedback jumps or scores drop
SafetyBlocked inputs, policy violations, data-leak checksAny confirmed leak or a spike in blocks
CostCost per request, per feature, per dayDaily spend exceeds budget or a single user consumes too much
SpeedTime to first token, total latency (median and slowest 5%)Latency climbs past your target
ReliabilityErrors, timeouts, rate limits, fallback usageError or fallback rate rises

When things go wrong

  1. Contain: switch the feature off with its flag, fall back to a safe path, or block the problem input.
  2. Investigate: use traces to find which prompt, model, document or tool caused it.
  3. Fix and verify: correct the cause, add the case to the evaluation set and re-run it.
  4. Communicate and learn: tell affected users if needed, and hold a blameless review.

Managing model changes

Providers release new models and retire old ones. A new model can be better overall but worse for your use case. Pin versions in production, watch provider deprecation notices, and treat any model change like a code change: run your evaluation set, compare cost and latency, shadow or gradually roll out, and keep the ability to roll back until the old version is retired.

What this means for your role

Developer

Ship with flags, tracing, cost tracking and fallbacks built in. Make model and prompt changes configuration you can roll back in minutes.

Tester / QA

Own post-release quality: weekly sampled reviews, feedback analysis and turning failures into new evaluation cases. Re-run evals before every model change.

Business Analyst / PO

Define the business signals of success and failure, such as resolution rate or edits needed, so dashboards show value, not just technical health.

Tech Lead / Architect

Standardise observability for all AI features: one trace format, shared dashboards, alert thresholds and a documented model upgrade process.

Project / Delivery Manager

Plan the rollout stages and success criteria for each stage, and budget ongoing time after launch for monitoring and improvement.

Scrum Master / Agile

Add a short weekly AI quality review to the team's rhythm: feedback, incidents, cost and the top improvement for next sprint.

Try it: write a launch and monitoring plan

  1. For an AI feature you know, list the rollout stages and the check that must pass before each step.
  2. Pick five metrics for the dashboard and an alert threshold for each.
  3. Write your three-step incident plan: how you switch it off, where you look first and who you inform.

Key takeaways

  • AI features keep changing after launch, so plan to observe and improve them continuously.
  • Release with feature flags, shadow mode and gradual rollouts, and prove value with A/B tests.
  • Trace every request with model, prompt version, retrieval, tools, tokens, cost, latency and feedback.
  • Alert on quality, safety, cost, speed and reliability; contain incidents fast; treat model changes like code changes.

Key terms

Feature flag
A switch that turns a feature on or off without redeploying.
Shadow mode
Running a new system on real traffic without showing its output, to compare results safely.
Gradual rollout
Releasing to a growing percentage of users with checks between steps.
Deprecation
A provider's notice that a model version will be retired.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. What does shadow mode do?

  2. 2. A user reports a bad answer. What do you need to investigate it?

  3. 3. Negative feedback suddenly doubles after a deploy. What is the first step?

  4. 4. The provider releases a newer model. How should you adopt it?

  5. 5. Why review a random sample of ordinary conversations, not just negative feedback?

Answer all 5 questions to check.

Practise more