Nishaglobal Education logo

Estimating and planning AI work

How to plan work whose outcome is uncertain: time-boxed experiments, the work people forget (data, evaluation, review, integration, change), milestones based on evidence, running-cost estimates, and how to communicate plans and progress to stakeholders.

Lesson 29 of 4416 min readQuiz: 5 questions

๐ŸŽฏ By the end of this lesson you will be able to:

  • Plan AI work with time-boxed experiments and evidence-based milestones
  • Include the often-forgotten work: data, evaluation, expert review, integration and adoption
  • Estimate running costs and communicate uncertainty honestly
In this lesson
  1. Planning when you don't know if it will work
  2. The work people forget
  3. Milestones based on evidence
  4. Estimating running costs
  5. Communicating plans and progress
  6. What this means for your role
  7. Key takeaways
  8. Quiz

Planning when you don't know if it will work

You can't promise "the AI will be 95% accurate by March" before you've measured anything. What you can promise is a time-boxed effort to find out, and a decision at the end. Plan the learning first, then plan the building once the evidence is in.

  • Time-box experiments: "two weeks to test whether retrieval finds the right policy for 80% of questions", not "build the policy assistant".
  • Fix time, flex scope: if quality isn't there, narrow the use case rather than extending the deadline indefinitely.
  • Plan in stages (Lesson 28): commit firmly to the next stage and roughly to the ones after.
  • Give ranges, not single dates, until evidence reduces the uncertainty.
Ideaยฑ wide rangeDiscoveryPoC resultsPilot resultsfirm estimateoptimisticpessimisticmost likely
Uncertainty narrows as evidence comes in. Estimates should be ranges early, becoming firmer after the proof of concept and pilot.

The work people forget

AI estimates often cover only the build: prompts, code, integration. In practice, much of the effort lies elsewhere.

Work itemWhat it involvesWho
Data access and preparationApprovals, extraction, cleaning, permissions, keeping documents currentData owners, engineers
Evaluation setCollecting and labelling realistic cases, agreeing rubricsDomain experts, evaluation engineer
Evaluation runs and iterationMany cycles of change, measure, improveEngineers, QA
Safety and red teamingAdversarial testing, guardrailsQA, security
Reviews and approvalsSecurity, privacy, legal, riskSpecialist teams
Integration and UXFitting into existing systems and screens, fallback when AI failsDevelopers, designers
Monitoring and operationsDashboards, alerts, runbooks, supportPlatform, operations
Change and trainingUpdated processes, user training, communicationBusiness owner, change team

Milestones based on evidence

Make milestones prove something, not just finish something. "Prompt written" says little; "85% usable on the 150-case evaluation set" says a lot.

Weak milestoneEvidence-based milestone
RAG pipeline builtCorrect document retrieved for โ‰ฅ 85% of evaluation questions
Pilot launchedPilot of 40 users running for 4 weeks with weekly quality and adoption report
Model integratedDrafts accepted with light edits โ‰ฅ 70% in the pilot; average handling time measured
Security doneRed team run with no critical findings open; approval recorded

Estimating running costs

A simple monthly cost estimate

Users: 400 agents ร— 30 drafts/day ร— 21 working days = 252,000 drafts/month
Tokens per draft: 3,000 input + 400 output (measured in the proof of concept)
Price (example): $1 per million input, $4 per million output tokens
Model cost: 252,000 ร— (3,000 ร— $1 + 400 ร— $4) / 1,000,000 โ‰ˆ $1,160/month
Add: evaluation runs, retrieval and hosting, monitoring, a safety margin of 30โ€“50%

Measured tokens ร— realistic volume ร— current prices, plus the extras. Re-check it in the pilot with real usage.

Prices are illustrative. Use your provider's current prices.

Communicating plans and progress

  • Show quality trends over time on the evaluation set, not just tasks completed.
  • State confidence explicitly: "likely", "uncertain until the pilot", with what would change it.
  • Report risks and decisions needed first, since stakeholders can act on those.
  • Avoid demo-driven expectations: show typical and failure cases, not only the best ones.

What this means for your role

Developer

Estimate in experiments and iterations, measure token usage early, and tell the plan owner as soon as quality looks stuck.

Tester / QA

Make evaluation work visible in the plan, with time for labelling, rubric agreement and repeated runs.

Business Analyst / PO

Bring volumes, peak times and current process timings, which drive both cost estimates and the value case.

Tech Lead / Architect

Identify integration, platform and approval work early, and estimate running cost and capacity per feature.

Project / Delivery Manager

Plan learning before building, use ranges and evidence-based milestones, include all the forgotten work, and secure expert time in writing.

Scrum Master / Agile

Help the team break AI work into small experiments with clear questions, and keep evaluation results visible in reviews.

Try it: re-plan an AI initiative

  1. Take an AI plan or idea and list every work item, using the forgotten-work table as a checklist.
  2. Rewrite three milestones so each proves something with a number.
  3. Make a rough monthly running-cost estimate using volume ร— tokens ร— price, plus a margin.

Key takeaways

  • Plan the learning first with time-boxed experiments; fix time and flex scope.
  • Include data, evaluation, review, approvals, integration, monitoring and change work, which often outweigh the core build.
  • Use evidence-based milestones and ranges that narrow as results come in.
  • Estimate running cost from measured tokens, realistic volume and current prices, and communicate uncertainty openly.

Key terms

Time-box
A fixed amount of time for a piece of work, after which you decide based on what you learned.
Cone of uncertainty
The idea that estimates are very uncertain early and become more precise as work progresses.
Evidence-based milestone
A milestone defined by a measured result rather than by completing an activity.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. Which is the better way to plan the first phase of an AI feature?

  2. 2. Which work is most often missing from AI estimates?

  3. 3. Which is an evidence-based milestone?

  4. 4. Quality is below target as the deadline approaches. What does "fix time, flex scope" suggest?

  5. 5. What should a running-cost estimate be based on?

Answer all 5 questions to check.

Practise more