Cost, speed and scale
How AI features are priced, a worked cost estimate, why some answers feel slow, and the levers teams use to make AI cheaper and faster: shorter context, the right model, caching, batching and limits.
🎯 By the end of this lesson you will be able to:
- Estimate the running cost of an AI feature from tokens and volume
- Explain what makes AI responses slow and how streaming helps
- Choose cost and speed levers: context size, model choice, caching, batching and output limits
In this lesson
How AI is priced
Most AI models used through an API are paid per token (Lesson 3). There is a price for input tokens, everything you send including instructions, history and documents, and a usually higher price for output tokens, what the model writes. Bigger, more capable models cost more per token than smaller ones. The bill for a feature is roughly tokens per request × price × number of requests.
A worked estimate
A support-reply assistant (illustrative prices)
Assume a model costs $3 per million input tokens and $15 per million output tokens. Each request sends 2,000 input tokens (instructions + retrieved policy + customer email) and gets 500 output tokens back. Volume: 10,000 requests a day.
Input: 2,000 × $3 ÷ 1,000,000 = $0.006 per request Output: 500 × $15 ÷ 1,000,000 = $0.0075 per request Total ≈ $0.0135 per request → about $135 a day → about $4,000 a month
Prices here are made up for the exercise and change often. Use your provider's current price list. The method is what matters.
Why some answers feel slow
- Output length: models write token by token, so long answers take longer.
- Model size: larger models are usually slower than smaller ones.
- Input size: very long prompts take longer to process before the first word appears.
- Multiple steps: chains and agents make several calls in a row, and the waiting adds up.
Two measures matter to users: time to first token (how long before anything appears) and total time. Streaming, showing the answer as it is written, makes waits feel much shorter even when the total time is the same. That is why chat assistants type their answers out live.
The levers for cost and speed
| Lever | What it means | Example |
|---|---|---|
| Send less context | Only the instructions and passages really needed | Retrieve 3 good chunks instead of 15 |
| Right-size the model | Use smaller, cheaper models for simple steps | Small model classifies; large model writes the reply |
| Cache | Reuse work that repeats | Prompt caching for a long fixed system prompt; saved answers to identical questions |
| Batch | Process non-urgent work in bulk | Overnight tagging of the day's tickets, often at a discount |
| Limit output | Ask for concise answers and set a maximum length | "Answer in under 120 words" |
Running at scale
- Rate limits: providers cap requests and tokens per minute. Plan for peaks and request higher limits early.
- Retries and fallbacks: calls sometimes fail or time out. Retry sensibly and have a fallback, such as a simpler model or a friendly "try again" message.
- Monitoring: track cost, speed, error rates and quality per feature, with budget alerts.
- Usage caps: limit how much one user or one agent run can consume, to prevent runaway bills.
What this means for your role
Developer
Log tokens, latency and cost for every call. Use streaming for user-facing answers, cache what repeats, set maximum output lengths and implement retries with backoff.
Tester / QA
Add performance checks: time to first token, total time and token counts for typical and worst-case inputs. Re-run quality evals whenever a cheaper model or shorter context is proposed.
Business Analyst / PO
Capture volume, response-time expectations and budget as requirements. "Answers within 3 seconds for 95% of requests" is testable; "fast" is not.
Tech Lead / Architect
Design for routing, caching and fallbacks from the start, and make cost per request a tracked architecture metric.
Project / Delivery Manager
Get a cost-per-request estimate and a monthly forecast before launch, set budget alerts, and review actual against forecast after release.
Scrum Master / Agile
Make AI cost and speed visible on the team's dashboard so trade-offs are discussed openly in planning.
Try it: cost your own idea
- Pick an AI feature idea and estimate input and output tokens per request (about 1.3 tokens per English word).
- Look up a current price for a small and a large model from a provider's pricing page.
- Calculate the monthly cost for both at your expected volume, and list two levers that would cut the cost.
Key takeaways
- Cost ≈ tokens per request × price × volume, with output tokens usually priced higher than input.
- Speed depends on output length, model size, input size and the number of steps; streaming improves perceived speed.
- The main levers are less context, right-sized models, caching, batching and output limits.
- At scale, plan for rate limits, retries, fallbacks, monitoring and usage caps, and check quality after every saving.
Key terms
- Latency
- How long a response takes, often measured as time to first token and total time.
- Streaming
- Showing a model's output as it is generated, token by token.
- Caching
- Storing and reusing results or repeated prompt parts to save cost and time.
- Rate limit
- A provider's cap on requests or tokens per minute.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. A request uses 1,000 input tokens at $2 per million and 200 output tokens at $10 per million. What does it cost?
2. What does streaming improve?
3. A RAG assistant sends 15 long chunks with every question. What lever most directly cuts cost and time?
4. What is model routing?
5. After switching to a cheaper model, what must you do?