Scaling and LLMOps
Running AI at organisation scale: the AI gateway in detail, routing and fallbacks, capacity and rate limits, caching, latency engineering, cost management and chargeback, version and deprecation management, and the platform team that keeps it all working.
๐ฏ By the end of this lesson you will be able to:
- Design an AI gateway with routing, limits, caching, fallbacks and observability
- Plan capacity, latency and cost for many AI features at once
- Set up the operating model: versions, deprecations, budgets and a platform team
In this lesson
What LLMOps means
LLMOps is the set of practices and tools for running language-model features reliably in production: deploying prompts and models, routing traffic, monitoring quality and cost, and managing change. One AI feature can be run by hand; twenty features across ten teams need shared infrastructure and clear operating practices.
The AI gateway
| Gateway capability | Why it matters |
|---|---|
| Authentication and keys | Provider keys never leave the gateway; each feature has its own identity |
| Routing | Send each request to the right model by task, cost tier, region or experiment |
| Rate limiting and quotas | Protect providers' limits and budgets per team, feature and user |
| Caching | Reuse identical results and repeated prompt prefixes to cut cost and latency |
| Fallbacks and retries | Retry with backoff; switch to a backup model or provider during outages |
| Guardrail hooks | Apply shared input and output checks consistently |
| Observability | Central logs, traces, token counts and cost per feature |
Capacity and rate limits
- Know each provider's limits on requests and tokens per minute, and request increases well before launches.
- Forecast demand per feature, including peaks such as Monday mornings or month-end.
- Separate interactive traffic from batch jobs, and give interactive traffic priority.
- For steady high volume, some providers offer reserved or provisioned capacity; compare its cost with pay-as-you-go.
- If self-hosting, plan GPU capacity, autoscaling and the time it takes to start new instances.
Latency engineering
- Stream user-facing responses (Lesson 12).
- Trim context: fewer, better retrieved chunks; summarise long histories.
- Right-size models: route simple tasks to faster models.
- Parallelise: run independent steps, such as retrieval and a classification call, at the same time.
- Cache carefully: exact-match caching is safe; caching "similar" questions by meaning can return a wrong answer for a subtly different question, so use it only where that risk is acceptable.
- Measure the slow tail: track the slowest 5% of requests, not just the average.
Cost management at scale
- Tag every request with feature, team and environment, so costs can be reported and charged back.
- Set budgets and alerts per feature and per team; cap spend per user and per agent run.
- Review the biggest cost drivers monthly: long contexts, chatty agents, oversized models, missing caching.
- Make evaluation spend visible too, and optimise it with caching and sampling (Lesson 23).
A monthly AI cost review
Top features by cost: Support copilot 46%, Document summariser 22%, Sales email drafts 14%, Others 18% Finding: summariser sends whole documents (avg 18,000 tokens) even for one-page answers Action: chunk and retrieve the relevant sections only; route summaries under 2 pages to a smaller model Expected: โ60% summariser cost, quality checked on the evaluation set before rollout
Cost data plus evaluation evidence turns "AI is expensive" into specific, safe savings.
Versions, deprecations and change
- Keep a catalogue of which features use which model versions, prompts and embedding models.
- Subscribe to provider deprecation notices and plan migrations early, with evaluation runs and gradual rollout (Lesson 18).
- Re-embedding a large document collection after an embedding model change takes time and money; schedule it.
- Use the same release discipline for prompts and configuration as for code: review, versioning and rollback.
The operating model
Many organisations create a small AI platform team that owns the gateway, retrieval services, evaluation tooling, observability and the approved model catalogue, while product teams own their features, prompts and evaluation sets. The platform team publishes standards, templates and office hours, so product teams don't have to become infrastructure experts.
What this means for your role
Developer
Route all model calls through the gateway with feature tags, use its caching and fallbacks, and keep prompts and model choices in versioned configuration.
Tester / QA
Include capacity, latency tail and fallback behaviour in pre-release testing, and re-run evaluations for every model migration.
Business Analyst / PO
Provide demand forecasts and peak patterns for each AI feature, so capacity and budgets can be planned.
Tech Lead / Architect
Design and own the gateway, the model and version catalogue, the cost model with chargeback, and the migration process for deprecations.
Project / Delivery Manager
Track AI cost per feature against value, own the budget, and plan migration work when providers retire models.
Scrum Master / Agile
Help platform and product teams agree service levels, request channels and release calendars, so shared AI infrastructure doesn't become a bottleneck.
Try it: design your gateway policy
- List three AI features and, for each, the model tier, latency target, monthly budget and fallback model.
- Write the routing rules and rate limits the gateway would apply to each.
- Pick one feature and list two cost drivers you would check first in a monthly review.
Key takeaways
- LLMOps is the practice of running AI features reliably: deployment, routing, monitoring, cost and change.
- An AI gateway centralises keys, routing, limits, caching, fallbacks, guardrail hooks and observability.
- Plan capacity and latency deliberately, cache carefully, and manage costs with tagging, budgets and monthly reviews.
- Track versions and deprecations, release prompts like code, and support product teams with a platform team.
Key terms
- LLMOps
- Practices and tooling for operating language-model applications in production.
- Chargeback
- Assigning shared infrastructure costs to the teams or features that use them.
- Provisioned capacity
- Reserved model throughput bought in advance, often for steady high volume.
- Semantic caching
- Reusing answers for questions judged similar in meaning, which risks wrong answers for subtly different questions.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Which of these is a gateway responsibility?
2. Why tag every AI request with feature and team?
3. What is the main risk of caching answers for questions that are merely similar in meaning?
4. A provider announces a model you use will be retired. What should you do?
5. Which latency metric best shows the worst user experience?