Nishaglobal Education logo

Measuring value and ROI

How to prove whether AI is worth it: linking AI quality to business outcomes, measuring a baseline before you start, comparing fairly with control groups, calculating ROI including all costs, and reporting results honestly.

Lesson 31 of 4416 min readQuiz: 5 questions

๐ŸŽฏ By the end of this lesson you will be able to:

  • Connect AI quality and adoption metrics to business outcomes
  • Measure a baseline and compare fairly using control groups or before-and-after data
  • Calculate ROI with all costs and report results honestly
In this lesson
  1. Why measuring value matters
  2. From AI quality to business outcomes
  3. Measure the baseline first
  4. Compare fairly
  5. Calculating ROI
  6. Reporting honestly
  7. What this means for your role
  8. Key takeaways
  9. Quiz

Why measuring value matters

"Our team loves the AI assistant" is nice but not enough to keep funding it. Leaders need to know whether AI saves time, reduces cost, improves quality or increases revenue, and by how much. Projects that can show value get scaled; those that can't are cut, even when they are actually useful.

From AI quality to business outcomes

Business outcomehandling time โˆ’20%, same qualityAdoptiondrafts used ยท accepted vs editedCostmodel ยท platform ยท peopleAI qualityeval score ยท groundedness ยท safetyROI = (value โˆ’ all costs)รท all costs
Business outcomes depend on adoption, which depends on quality. Measure all three layers to understand why value is or isn't appearing.
LayerExample metrics
Business outcomesHandling time, cost per case, revenue, customer satisfaction, errors, compliance findings
Adoption and usageActive users, how often drafts are used, accepted vs edited vs rejected, repeat use
AI qualityEvaluation scores, accuracy, groundedness, safety, latency
CostModel and infrastructure cost, team cost, support cost

Measure the baseline first

You can't show improvement without knowing where you started. Before launch, measure the current process: how long tasks take, error rates, volumes and costs. Many AI projects skip this and later can't prove anything.

  • Use existing system data where possible (ticket times, case volumes), as it is more reliable than surveys.
  • Measure over a long enough period to cover normal variation, such as several weeks.
  • Agree the definitions: what exactly counts as "handling time" or an "error".

Compare fairly

MethodHow it worksStrength
Control group (A/B)Some users get AI, similar users don't, same periodStrongest evidence; rules out seasonal changes
Staggered rolloutTeams adopt at different times; compare as each switchesPractical when everyone will get it eventually
Before and afterCompare the same team before and after launchSimple, but other changes can distort it
Self-reported time savedAsk users how much time they saveEasy, but often overestimates

Calculating ROI

A simple ROI calculation

Pilot: 40 support agents with the AI drafting assistant vs 40 similar agents without, over 8 weeks
Result: average handling time 9.0 โ†’ 7.2 minutes (โˆ’20%), quality scores unchanged
Scale: 400 agents ร— 30 cases/day ร— 1.8 min saved ร— 21 days โ‰ˆ 7,560 hours/month
Value: 7,560 h ร— $30/h โ‰ˆ $227,000/month in capacity
Costs: model + platform โ‰ˆ $6,000/month; team and support โ‰ˆ $40,000/month
ROI: (227,000 โˆ’ 46,000) / 46,000 โ‰ˆ 3.9ร— per month

Measured against a control group, with all costs included, and stated as capacity freed rather than guaranteed savings.

Time saved only becomes money saved if it is used for something valuable or reduces costs. Say what will happen to the freed capacity.

Reporting honestly

  • Report ranges and confidence, not just the best number.
  • Include what didn't work and where AI made things worse.
  • Show quality and safety alongside value, since savings that come with more errors are not real savings.
  • Re-measure regularly: value can fade as models, processes and users change.

What this means for your role

Developer

Instrument features to log usage, acceptance and edits from day one, so value can be measured without extra projects later.

Tester / QA

Make sure quality is tracked alongside value; a faster process that produces more errors is a regression.

Business Analyst / PO

Define the business metrics and baseline in discovery, with precise definitions agreed with the business owner.

Tech Lead / Architect

Design the system so control groups, feature flags and analytics are possible without major changes.

Project / Delivery Manager

Own the value case: baseline, comparison method, ROI with all costs and honest regular reporting to sponsors.

Scrum Master / Agile

Bring outcome data into reviews and retrospectives, so the team optimises for value, not just features shipped.

Try it: build a value case

  1. Pick an AI use case and write one business outcome, two adoption metrics and two quality metrics for it.
  2. Describe how you'd measure the baseline and which comparison method you'd use.
  3. Sketch the ROI calculation, including all costs and what happens to time saved.

Key takeaways

  • Measure value in layers: AI quality drives adoption, which drives business outcomes.
  • Measure the baseline before launch, with agreed definitions.
  • Compare fairly, ideally with a control group, and watch for novelty effects and hidden checking time.
  • Calculate ROI with all costs, explain what happens to time saved, and report honestly with ranges.

Key terms

Baseline
Measurements of the current process before a change, used for comparison.
Control group
Similar users who don't get the change, used to isolate its effect.
ROI
Return on investment: the benefit gained relative to the cost.
Novelty effect
A temporary boost in usage or performance because something is new.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. What should be measured before launching an AI feature?

  2. 2. Which comparison gives the strongest evidence of AI impact?

  3. 3. Quality scores are high but few people use the AI feature. What does this suggest?

  4. 4. Which cost is often forgotten when estimating time saved?

  5. 5. Why re-measure value regularly after launch?

Answer all 5 questions to check.

Practise more