Shipping and monitoring AI features
Releasing AI features safely with flags, shadow runs and gradual rollouts, tracing every request, collecting user feedback, building dashboards and alerts for quality, cost and speed, and handling incidents and model changes.
🎯 By the end of this lesson you will be able to:
- Plan a safe release using feature flags, shadow mode and gradual rollout
- Define what to log and monitor for AI features: quality, cost, latency, errors and safety
- Respond to AI incidents and manage provider model updates without surprises
In this lesson
Launch is the start, not the finish
Normal software mostly behaves the same after release as in testing. AI features keep surprising you: real users ask things your evaluation set never imagined, documents change, costs move with usage, and providers update models. Successful teams treat AI features as living systems they watch and improve every week.
Releasing safely
| Technique | How it works | Why |
|---|---|---|
| Feature flag | Turn the feature on or off without a deploy | Instant rollback and targeted access |
| Shadow mode | The AI runs on real traffic but its output isn't shown; compare with what happened | Measure quality on real inputs with zero user risk |
| Internal beta | Staff use it first | Fast feedback from forgiving users |
| Gradual rollout | 1%, then 10%, then 50%, then everyone, with checks between steps | Limits the impact of problems |
| A/B test | Compare against the old experience or another version | Proves real-world value, not just demo quality |
Trace every request
When a user reports a bad answer, you need to see exactly what happened. Log a trace for each AI request, protecting personal data as Lesson 17 describes:
- Request ID, user or tenant (pseudonymised where possible), feature and timestamp
- Model and version, prompt template version and key settings
- Retrieved documents or chunk IDs, and tool calls with inputs and results
- Input and output token counts, cost, latency and time to first token
- Validation results, errors, retries and fallbacks used
- User feedback attached later, such as thumbs up or down, edits or escalation
Feedback: your best source of new test cases
Make feedback easy: thumbs up or down with an optional comment, "report a problem", and implicit signals such as users editing the AI's draft heavily, retrying or asking for a human. Review negative feedback every week, find the patterns, fix the causes and add the examples to your evaluation set (Lesson 11).
Dashboards and alerts
| Area | Watch | Alert when |
|---|---|---|
| Quality | Feedback rate, sampled rubric scores, escalation rate | Negative feedback jumps or scores drop |
| Safety | Blocked inputs, policy violations, data-leak checks | Any confirmed leak or a spike in blocks |
| Cost | Cost per request, per feature, per day | Daily spend exceeds budget or a single user consumes too much |
| Speed | Time to first token, total latency (median and slowest 5%) | Latency climbs past your target |
| Reliability | Errors, timeouts, rate limits, fallback usage | Error or fallback rate rises |
When things go wrong
- Contain: switch the feature off with its flag, fall back to a safe path, or block the problem input.
- Investigate: use traces to find which prompt, model, document or tool caused it.
- Fix and verify: correct the cause, add the case to the evaluation set and re-run it.
- Communicate and learn: tell affected users if needed, and hold a blameless review.
Managing model changes
Providers release new models and retire old ones. A new model can be better overall but worse for your use case. Pin versions in production, watch provider deprecation notices, and treat any model change like a code change: run your evaluation set, compare cost and latency, shadow or gradually roll out, and keep the ability to roll back until the old version is retired.
What this means for your role
Developer
Ship with flags, tracing, cost tracking and fallbacks built in. Make model and prompt changes configuration you can roll back in minutes.
Tester / QA
Own post-release quality: weekly sampled reviews, feedback analysis and turning failures into new evaluation cases. Re-run evals before every model change.
Business Analyst / PO
Define the business signals of success and failure, such as resolution rate or edits needed, so dashboards show value, not just technical health.
Tech Lead / Architect
Standardise observability for all AI features: one trace format, shared dashboards, alert thresholds and a documented model upgrade process.
Project / Delivery Manager
Plan the rollout stages and success criteria for each stage, and budget ongoing time after launch for monitoring and improvement.
Scrum Master / Agile
Add a short weekly AI quality review to the team's rhythm: feedback, incidents, cost and the top improvement for next sprint.
Try it: write a launch and monitoring plan
- For an AI feature you know, list the rollout stages and the check that must pass before each step.
- Pick five metrics for the dashboard and an alert threshold for each.
- Write your three-step incident plan: how you switch it off, where you look first and who you inform.
Key takeaways
- AI features keep changing after launch, so plan to observe and improve them continuously.
- Release with feature flags, shadow mode and gradual rollouts, and prove value with A/B tests.
- Trace every request with model, prompt version, retrieval, tools, tokens, cost, latency and feedback.
- Alert on quality, safety, cost, speed and reliability; contain incidents fast; treat model changes like code changes.
Key terms
- Feature flag
- A switch that turns a feature on or off without redeploying.
- Shadow mode
- Running a new system on real traffic without showing its output, to compare results safely.
- Gradual rollout
- Releasing to a growing percentage of users with checks between steps.
- Deprecation
- A provider's notice that a model version will be retired.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. What does shadow mode do?
2. A user reports a bad answer. What do you need to investigate it?
3. Negative feedback suddenly doubles after a deploy. What is the first step?
4. The provider releases a newer model. How should you adopt it?
5. Why review a random sample of ordinary conversations, not just negative feedback?