Measuring value and ROI
How to prove whether AI is worth it: linking AI quality to business outcomes, measuring a baseline before you start, comparing fairly with control groups, calculating ROI including all costs, and reporting results honestly.
๐ฏ By the end of this lesson you will be able to:
- Connect AI quality and adoption metrics to business outcomes
- Measure a baseline and compare fairly using control groups or before-and-after data
- Calculate ROI with all costs and report results honestly
In this lesson
Why measuring value matters
"Our team loves the AI assistant" is nice but not enough to keep funding it. Leaders need to know whether AI saves time, reduces cost, improves quality or increases revenue, and by how much. Projects that can show value get scaled; those that can't are cut, even when they are actually useful.
From AI quality to business outcomes
| Layer | Example metrics |
|---|---|
| Business outcomes | Handling time, cost per case, revenue, customer satisfaction, errors, compliance findings |
| Adoption and usage | Active users, how often drafts are used, accepted vs edited vs rejected, repeat use |
| AI quality | Evaluation scores, accuracy, groundedness, safety, latency |
| Cost | Model and infrastructure cost, team cost, support cost |
Measure the baseline first
You can't show improvement without knowing where you started. Before launch, measure the current process: how long tasks take, error rates, volumes and costs. Many AI projects skip this and later can't prove anything.
- Use existing system data where possible (ticket times, case volumes), as it is more reliable than surveys.
- Measure over a long enough period to cover normal variation, such as several weeks.
- Agree the definitions: what exactly counts as "handling time" or an "error".
Compare fairly
| Method | How it works | Strength |
|---|---|---|
| Control group (A/B) | Some users get AI, similar users don't, same period | Strongest evidence; rules out seasonal changes |
| Staggered rollout | Teams adopt at different times; compare as each switches | Practical when everyone will get it eventually |
| Before and after | Compare the same team before and after launch | Simple, but other changes can distort it |
| Self-reported time saved | Ask users how much time they save | Easy, but often overestimates |
Calculating ROI
A simple ROI calculation
Pilot: 40 support agents with the AI drafting assistant vs 40 similar agents without, over 8 weeks Result: average handling time 9.0 โ 7.2 minutes (โ20%), quality scores unchanged Scale: 400 agents ร 30 cases/day ร 1.8 min saved ร 21 days โ 7,560 hours/month Value: 7,560 h ร $30/h โ $227,000/month in capacity Costs: model + platform โ $6,000/month; team and support โ $40,000/month ROI: (227,000 โ 46,000) / 46,000 โ 3.9ร per month
Measured against a control group, with all costs included, and stated as capacity freed rather than guaranteed savings.
Time saved only becomes money saved if it is used for something valuable or reduces costs. Say what will happen to the freed capacity.
Reporting honestly
- Report ranges and confidence, not just the best number.
- Include what didn't work and where AI made things worse.
- Show quality and safety alongside value, since savings that come with more errors are not real savings.
- Re-measure regularly: value can fade as models, processes and users change.
What this means for your role
Developer
Instrument features to log usage, acceptance and edits from day one, so value can be measured without extra projects later.
Tester / QA
Make sure quality is tracked alongside value; a faster process that produces more errors is a regression.
Business Analyst / PO
Define the business metrics and baseline in discovery, with precise definitions agreed with the business owner.
Tech Lead / Architect
Design the system so control groups, feature flags and analytics are possible without major changes.
Project / Delivery Manager
Own the value case: baseline, comparison method, ROI with all costs and honest regular reporting to sponsors.
Scrum Master / Agile
Bring outcome data into reviews and retrospectives, so the team optimises for value, not just features shipped.
Try it: build a value case
- Pick an AI use case and write one business outcome, two adoption metrics and two quality metrics for it.
- Describe how you'd measure the baseline and which comparison method you'd use.
- Sketch the ROI calculation, including all costs and what happens to time saved.
Key takeaways
- Measure value in layers: AI quality drives adoption, which drives business outcomes.
- Measure the baseline before launch, with agreed definitions.
- Compare fairly, ideally with a control group, and watch for novelty effects and hidden checking time.
- Calculate ROI with all costs, explain what happens to time saved, and report honestly with ranges.
Key terms
- Baseline
- Measurements of the current process before a change, used for comparison.
- Control group
- Similar users who don't get the change, used to isolate its effect.
- ROI
- Return on investment: the benefit gained relative to the cost.
- Novelty effect
- A temporary boost in usage or performance because something is new.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. What should be measured before launching an AI feature?
2. Which comparison gives the strongest evidence of AI impact?
3. Quality scores are high but few people use the AI feature. What does this suggest?
4. Which cost is often forgotten when estimating time saved?
5. Why re-measure value regularly after launch?