The AI project lifecycle
How AI projects move from idea to discovery, proof of concept, pilot and production, what makes them different from ordinary software projects, the go/no-go gates between stages, and the common reasons AI projects stall in "pilot purgatory".
๐ฏ By the end of this lesson you will be able to:
- Explain how AI projects differ from traditional software projects
- Run an AI project through clear stages with evidence-based go/no-go gates
- Spot and prevent the common reasons AI projects stall before production
In this lesson
Why AI projects are different
In a traditional project, if the team builds what was specified, it works. In an AI project, you can build everything correctly and still find the AI isn't good enough for the job. Quality is uncertain until you measure it with real data. That single fact changes how you plan, report and make decisions.
| Traditional software | AI features |
|---|---|
| Behaviour is specified and predictable | Behaviour is probabilistic; quality is measured, not assumed |
| Done means the requirements are met | Done means quality thresholds are met on an evaluation set |
| Risk is mostly scope, time and budget | Plus quality risk, data risk, model change and misuse |
| After launch, mainly bug fixes | After launch, ongoing monitoring, evaluation and tuning |
| Cost mostly fixed (build, hosting) | Running cost grows with usage (tokens, calls) |
The five stages
| Stage | Goal | Typical length | Exit evidence |
|---|---|---|---|
| 1. Discovery | Is this a real problem worth solving with AI? | 1โ3 weeks | Problem statement, users, baseline numbers, data available, success metric |
| 2. Proof of concept | Can AI do the core task well enough? | 2โ6 weeks | Results on a small evaluation set vs a target |
| 3. Pilot | Does it work for real users in real conditions? | 1โ3 months | Quality, adoption, feedback and cost from a limited group |
| 4. Production | Run it reliably for everyone | Ongoing | Monitoring, support, runbooks, approvals in place |
| 5. Operate and improve | Keep value high as things change | Ongoing | Regular quality, cost and value reviews |
Go/no-go gates
Agree the gate criteria before each stage starts, with numbers. Otherwise every result looks "promising" and projects drift on without a decision. A gate can say go, stop, or change direction (pivot), and stopping a project early is a success when the evidence says it won't work.
Gate criteria for a proof of concept
Use case: draft replies to customer emails about deliveries Go if: โฅ 80% of drafts rated "usable with light edits" on 150 real past emails, zero drafts with wrong refund amounts, cost per draft < $0.02 Pivot if: 60โ80% usable โ narrow scope to the 3 most common email types Stop if: < 60% usable, or the order data needed isn't accessible
Everyone knows in advance what success looks like and what happens next.
Why AI projects stall
Many AI projects produce an impressive demo and then never reach production. This is sometimes called pilot purgatory. The usual causes are predictable, so you can plan against them.
| Cause | Prevention |
|---|---|
| No clear business owner or success metric | Name an owner and a measurable outcome in discovery |
| Demo built on a handful of hand-picked examples | Measure on a realistic evaluation set from the start |
| Data access, quality or permissions blocked late | Check data access and approvals in discovery |
| Security, legal or risk review starts at the end | Involve them at the first gate, scaled to risk |
| No plan for integration, support and monitoring | Plan production work from the pilot stage |
| Users don't change how they work | Involve users early; plan training and workflow change |
| Running cost unknown until scale | Estimate cost per use in the proof of concept |
The people you need
- Business owner: owns the problem, the success metric and the go/no-go decisions.
- Domain experts: define good answers, label evaluation data and review outputs.
- AI engineers and developers: build, integrate and tune.
- Evaluation engineer or QA: builds the evaluation set and measures quality.
- Architect: platform, security and integration choices.
- Risk, security, legal and privacy: approvals proportional to risk.
- Change and training: help users adopt the new way of working.
What this means for your role
Developer
Build the proof of concept to answer the gate question cheaply. Don't over-engineer it, but measure it honestly on realistic data.
Tester / QA
Create the evaluation set during discovery or early in the proof of concept; gates depend on it.
Business Analyst / PO
Own the problem statement, baseline numbers and the success metric in discovery, and keep them visible at every gate.
Tech Lead / Architect
Flag architecture, data access and security blockers in discovery, and design the pilot so it can grow into production.
Project / Delivery Manager
Run the project in stages with agreed gate criteria, report evidence rather than optimism, and treat an early stop as a valid outcome.
Scrum Master / Agile
Help the team work in short experiment cycles within each stage, and make the gate criteria part of the team's shared definition of done.
Try it: write gate criteria
- Pick an AI idea in your organisation that is at idea or demo stage.
- Write go, pivot and stop criteria for its proof of concept, with numbers for quality, safety and cost.
- List the three biggest reasons it could stall later, and one action to prevent each now.
Key takeaways
- AI quality is uncertain until measured, so AI projects run in stages with evidence-based gates.
- The five stages: discovery, proof of concept, pilot, production, operate and improve.
- Agree go, pivot and stop criteria with numbers before each stage starts; stopping early is a valid outcome.
- Prevent pilot purgatory with an owner, a metric, realistic data, early approvals, production planning and user adoption.
Key terms
- Proof of concept (PoC)
- A small, quick test of whether the core idea can work at all.
- Pilot
- A limited real-world rollout to a small group of users to test value and practicality.
- Go/no-go gate
- A decision point where evidence decides whether to continue, change direction or stop.
- Pilot purgatory
- When AI projects get stuck after a promising pilot and never reach production.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. What is the biggest difference between AI projects and traditional software projects?
2. When should go/no-go criteria for a stage be agreed?
3. A proof of concept looked great on 10 hand-picked examples. What is the risk?
4. When should security, legal and risk teams be involved?
5. The proof of concept clearly misses its targets. What is a good outcome?