Nishaglobal Education logo

Scaling and LLMOps

Running AI at organisation scale: the AI gateway in detail, routing and fallbacks, capacity and rate limits, caching, latency engineering, cost management and chargeback, version and deprecation management, and the platform team that keeps it all working.

Lesson 27 of 4417 min readQuiz: 5 questions

๐ŸŽฏ By the end of this lesson you will be able to:

  • Design an AI gateway with routing, limits, caching, fallbacks and observability
  • Plan capacity, latency and cost for many AI features at once
  • Set up the operating model: versions, deprecations, budgets and a platform team
In this lesson
  1. What LLMOps means
  2. The AI gateway
  3. Capacity and rate limits
  4. Latency engineering
  5. Cost management at scale
  6. Versions, deprecations and change
  7. The operating model
  8. What this means for your role
  9. Key takeaways
  10. Quiz

What LLMOps means

LLMOps is the set of practices and tools for running language-model features reliably in production: deploying prompts and models, routing traffic, monitoring quality and cost, and managing change. One AI feature can be run by hand; twenty features across ten teams need shared infrastructure and clear operating practices.

The AI gateway

Feature AFeature BFeature CBatch jobsAI gatewayauth + keysquotas + rate limitsguardrail hookscacherouting + fallbackslogs, tokens, costLarge modelcomplex tasksSmall modelsimple, fastBackup providerduring outages
Every feature calls the gateway; the gateway applies policy, routes to the right model, caches, falls back during outages and records everything.
Gateway capabilityWhy it matters
Authentication and keysProvider keys never leave the gateway; each feature has its own identity
RoutingSend each request to the right model by task, cost tier, region or experiment
Rate limiting and quotasProtect providers' limits and budgets per team, feature and user
CachingReuse identical results and repeated prompt prefixes to cut cost and latency
Fallbacks and retriesRetry with backoff; switch to a backup model or provider during outages
Guardrail hooksApply shared input and output checks consistently
ObservabilityCentral logs, traces, token counts and cost per feature

Capacity and rate limits

  • Know each provider's limits on requests and tokens per minute, and request increases well before launches.
  • Forecast demand per feature, including peaks such as Monday mornings or month-end.
  • Separate interactive traffic from batch jobs, and give interactive traffic priority.
  • For steady high volume, some providers offer reserved or provisioned capacity; compare its cost with pay-as-you-go.
  • If self-hosting, plan GPU capacity, autoscaling and the time it takes to start new instances.

Latency engineering

  • Stream user-facing responses (Lesson 12).
  • Trim context: fewer, better retrieved chunks; summarise long histories.
  • Right-size models: route simple tasks to faster models.
  • Parallelise: run independent steps, such as retrieval and a classification call, at the same time.
  • Cache carefully: exact-match caching is safe; caching "similar" questions by meaning can return a wrong answer for a subtly different question, so use it only where that risk is acceptable.
  • Measure the slow tail: track the slowest 5% of requests, not just the average.

Cost management at scale

  • Tag every request with feature, team and environment, so costs can be reported and charged back.
  • Set budgets and alerts per feature and per team; cap spend per user and per agent run.
  • Review the biggest cost drivers monthly: long contexts, chatty agents, oversized models, missing caching.
  • Make evaluation spend visible too, and optimise it with caching and sampling (Lesson 23).

A monthly AI cost review

Top features by cost: Support copilot 46%, Document summariser 22%, Sales email drafts 14%, Others 18%
Finding: summariser sends whole documents (avg 18,000 tokens) even for one-page answers
Action: chunk and retrieve the relevant sections only; route summaries under 2 pages to a smaller model
Expected: โˆ’60% summariser cost, quality checked on the evaluation set before rollout

Cost data plus evaluation evidence turns "AI is expensive" into specific, safe savings.

Versions, deprecations and change

  • Keep a catalogue of which features use which model versions, prompts and embedding models.
  • Subscribe to provider deprecation notices and plan migrations early, with evaluation runs and gradual rollout (Lesson 18).
  • Re-embedding a large document collection after an embedding model change takes time and money; schedule it.
  • Use the same release discipline for prompts and configuration as for code: review, versioning and rollback.

The operating model

Many organisations create a small AI platform team that owns the gateway, retrieval services, evaluation tooling, observability and the approved model catalogue, while product teams own their features, prompts and evaluation sets. The platform team publishes standards, templates and office hours, so product teams don't have to become infrastructure experts.

What this means for your role

Developer

Route all model calls through the gateway with feature tags, use its caching and fallbacks, and keep prompts and model choices in versioned configuration.

Tester / QA

Include capacity, latency tail and fallback behaviour in pre-release testing, and re-run evaluations for every model migration.

Business Analyst / PO

Provide demand forecasts and peak patterns for each AI feature, so capacity and budgets can be planned.

Tech Lead / Architect

Design and own the gateway, the model and version catalogue, the cost model with chargeback, and the migration process for deprecations.

Project / Delivery Manager

Track AI cost per feature against value, own the budget, and plan migration work when providers retire models.

Scrum Master / Agile

Help platform and product teams agree service levels, request channels and release calendars, so shared AI infrastructure doesn't become a bottleneck.

Try it: design your gateway policy

  1. List three AI features and, for each, the model tier, latency target, monthly budget and fallback model.
  2. Write the routing rules and rate limits the gateway would apply to each.
  3. Pick one feature and list two cost drivers you would check first in a monthly review.

Key takeaways

  • LLMOps is the practice of running AI features reliably: deployment, routing, monitoring, cost and change.
  • An AI gateway centralises keys, routing, limits, caching, fallbacks, guardrail hooks and observability.
  • Plan capacity and latency deliberately, cache carefully, and manage costs with tagging, budgets and monthly reviews.
  • Track versions and deprecations, release prompts like code, and support product teams with a platform team.

Key terms

LLMOps
Practices and tooling for operating language-model applications in production.
Chargeback
Assigning shared infrastructure costs to the teams or features that use them.
Provisioned capacity
Reserved model throughput bought in advance, often for steady high volume.
Semantic caching
Reusing answers for questions judged similar in meaning, which risks wrong answers for subtly different questions.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. Which of these is a gateway responsibility?

  2. 2. Why tag every AI request with feature and team?

  3. 3. What is the main risk of caching answers for questions that are merely similar in meaning?

  4. 4. A provider announces a model you use will be retired. What should you do?

  5. 5. Which latency metric best shows the worst user experience?

Answer all 5 questions to check.

Practise more