Estimating and planning AI work
How to plan work whose outcome is uncertain: time-boxed experiments, the work people forget (data, evaluation, review, integration, change), milestones based on evidence, running-cost estimates, and how to communicate plans and progress to stakeholders.
๐ฏ By the end of this lesson you will be able to:
- Plan AI work with time-boxed experiments and evidence-based milestones
- Include the often-forgotten work: data, evaluation, expert review, integration and adoption
- Estimate running costs and communicate uncertainty honestly
In this lesson
Planning when you don't know if it will work
You can't promise "the AI will be 95% accurate by March" before you've measured anything. What you can promise is a time-boxed effort to find out, and a decision at the end. Plan the learning first, then plan the building once the evidence is in.
- Time-box experiments: "two weeks to test whether retrieval finds the right policy for 80% of questions", not "build the policy assistant".
- Fix time, flex scope: if quality isn't there, narrow the use case rather than extending the deadline indefinitely.
- Plan in stages (Lesson 28): commit firmly to the next stage and roughly to the ones after.
- Give ranges, not single dates, until evidence reduces the uncertainty.
The work people forget
AI estimates often cover only the build: prompts, code, integration. In practice, much of the effort lies elsewhere.
| Work item | What it involves | Who |
|---|---|---|
| Data access and preparation | Approvals, extraction, cleaning, permissions, keeping documents current | Data owners, engineers |
| Evaluation set | Collecting and labelling realistic cases, agreeing rubrics | Domain experts, evaluation engineer |
| Evaluation runs and iteration | Many cycles of change, measure, improve | Engineers, QA |
| Safety and red teaming | Adversarial testing, guardrails | QA, security |
| Reviews and approvals | Security, privacy, legal, risk | Specialist teams |
| Integration and UX | Fitting into existing systems and screens, fallback when AI fails | Developers, designers |
| Monitoring and operations | Dashboards, alerts, runbooks, support | Platform, operations |
| Change and training | Updated processes, user training, communication | Business owner, change team |
Milestones based on evidence
Make milestones prove something, not just finish something. "Prompt written" says little; "85% usable on the 150-case evaluation set" says a lot.
| Weak milestone | Evidence-based milestone |
|---|---|
| RAG pipeline built | Correct document retrieved for โฅ 85% of evaluation questions |
| Pilot launched | Pilot of 40 users running for 4 weeks with weekly quality and adoption report |
| Model integrated | Drafts accepted with light edits โฅ 70% in the pilot; average handling time measured |
| Security done | Red team run with no critical findings open; approval recorded |
Estimating running costs
A simple monthly cost estimate
Users: 400 agents ร 30 drafts/day ร 21 working days = 252,000 drafts/month Tokens per draft: 3,000 input + 400 output (measured in the proof of concept) Price (example): $1 per million input, $4 per million output tokens Model cost: 252,000 ร (3,000 ร $1 + 400 ร $4) / 1,000,000 โ $1,160/month Add: evaluation runs, retrieval and hosting, monitoring, a safety margin of 30โ50%
Measured tokens ร realistic volume ร current prices, plus the extras. Re-check it in the pilot with real usage.
Prices are illustrative. Use your provider's current prices.
Communicating plans and progress
- Show quality trends over time on the evaluation set, not just tasks completed.
- State confidence explicitly: "likely", "uncertain until the pilot", with what would change it.
- Report risks and decisions needed first, since stakeholders can act on those.
- Avoid demo-driven expectations: show typical and failure cases, not only the best ones.
What this means for your role
Developer
Estimate in experiments and iterations, measure token usage early, and tell the plan owner as soon as quality looks stuck.
Tester / QA
Make evaluation work visible in the plan, with time for labelling, rubric agreement and repeated runs.
Business Analyst / PO
Bring volumes, peak times and current process timings, which drive both cost estimates and the value case.
Tech Lead / Architect
Identify integration, platform and approval work early, and estimate running cost and capacity per feature.
Project / Delivery Manager
Plan learning before building, use ranges and evidence-based milestones, include all the forgotten work, and secure expert time in writing.
Scrum Master / Agile
Help the team break AI work into small experiments with clear questions, and keep evaluation results visible in reviews.
Try it: re-plan an AI initiative
- Take an AI plan or idea and list every work item, using the forgotten-work table as a checklist.
- Rewrite three milestones so each proves something with a number.
- Make a rough monthly running-cost estimate using volume ร tokens ร price, plus a margin.
Key takeaways
- Plan the learning first with time-boxed experiments; fix time and flex scope.
- Include data, evaluation, review, approvals, integration, monitoring and change work, which often outweigh the core build.
- Use evidence-based milestones and ranges that narrow as results come in.
- Estimate running cost from measured tokens, realistic volume and current prices, and communicate uncertainty openly.
Key terms
- Time-box
- A fixed amount of time for a piece of work, after which you decide based on what you learned.
- Cone of uncertainty
- The idea that estimates are very uncertain early and become more precise as work progresses.
- Evidence-based milestone
- A milestone defined by a measured result rather than by completing an activity.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Which is the better way to plan the first phase of an AI feature?
2. Which work is most often missing from AI estimates?
3. Which is an evidence-based milestone?
4. Quality is below target as the deadline approaches. What does "fix time, flex scope" suggest?
5. What should a running-cost estimate be based on?