Your first LLM-powered feature
The architecture of a real AI feature, what goes into a model API call, getting validated structured output, handling errors, timeouts and rate limits, keeping keys safe, and testing it properly.
๐ฏ By the end of this lesson you will be able to:
- Sketch the architecture of an AI feature and explain why the model is called from your backend
- Describe the parts of a model API request and response
- Build in validation, error handling, streaming and tests from the start
In this lesson
The architecture of a simple AI feature
Most AI features follow the same shape. The user interacts with your app. Your backend builds the prompt from a template, the user's input and any data it needs, then calls the model provider's API. It checks the response, stores what it needs for monitoring, and returns a result to the user. The model is a service you call, much like a payment or email API.
What goes into a model API call
Provider APIs differ in detail, but almost all chat-style calls contain the same parts. The example below is simplified pseudo-code, not any specific SDK.
A typical request (pseudo-code)
response = model_api.create({
model: "<model name, pinned to a version>",
system: load_template("ticket-triage", version="v3"),
messages: [{ role: "user", content: ticket_text }],
max_output_tokens: 300,
temperature: 0,
response_format: TRIAGE_JSON_SCHEMA,
timeout_seconds: 20
})The response contains the generated content (here, JSON), token counts for input and output, a stop reason (finished, hit the token limit, or asked to call a tool) and an ID you should log.
| Part | Why it matters |
|---|---|
| Model and version | Pin a version so behaviour doesn't change under you; upgrade deliberately after re-running evals |
| System prompt | Stable instructions loaded from a versioned template |
| Messages | The user input and any retrieved context, clearly separated |
| Max output tokens | Caps cost and latency; check the stop reason for truncated answers |
| Temperature | Low for extraction and classification, higher for creative text |
| Response format / schema | Asks for, or enforces, structured output |
| Timeout | A slow call shouldn't hang your app |
Validate everything that comes back
Treat model output like input from an untrusted user. Parse it, validate it against your schema, check allowed values and lengths, and decide what to do when it fails: retry once with the error message, fall back to a safe default, or send the case to a person.
Validation flow (pseudo-code)
result = parse_json(response.content)
if not TRIAGE_SCHEMA.is_valid(result):
result = retry_once_with_feedback(errors)
if not TRIAGE_SCHEMA.is_valid(result):
return route_to_human(ticket)
if result.category not in ALLOWED_CATEGORIES:
result.category = "other"The feature keeps working even when the model misbehaves, and bad outputs never reach your database unchecked.
Errors, timeouts and rate limits
- Timeouts: set them on every call, and show the user a helpful message if one fires.
- Rate limits: providers return a "too many requests" error (HTTP 429). Retry with exponential backoff and jitter, and queue non-urgent work.
- Transient errors: retry a small number of times; don't retry errors caused by a bad request.
- Truncation: if the stop reason says the token limit was hit, the answer may be cut off. Handle it rather than showing half a sentence.
- Fallbacks: a secondary model, a cached answer or a simple non-AI path keeps the feature usable during an outage.
User experience: streaming and honesty
For text users read, stream the response so words appear immediately. Show that the content is AI-generated where users rely on it, offer a way to report problems, and make it easy to edit or undo AI suggestions. For structured tasks, such as tagging or routing, streaming is unnecessary. Return the validated result instead.
Testing an AI feature
| Test type | What it checks | Runs |
|---|---|---|
| Unit tests with a mocked model | Prompt building, parsing, validation, retries, fallbacks | Every commit, fast and free |
| Evaluation set (Lesson 11) | Real quality on realistic inputs with the real model | On prompt, model or retrieval changes |
| Contract tests | Your code still handles the provider's response format | Regularly and on SDK upgrades |
| Load and cost tests | Latency, rate limits and cost at expected volume | Before launch and major changes |
What this means for your role
Developer
Build a thin, well-tested wrapper around model calls: versioned prompts, schema validation, retries with backoff, timeouts, streaming and structured logging of model, prompt version, tokens and latency.
Tester / QA
Mock the model to test error paths: invalid JSON, wrong values, timeouts, 429s and truncated output. Then run the evaluation set against the real model for quality.
Business Analyst / PO
Specify what happens when the AI fails or is unsure: the fallback, the message users see, and when a person takes over.
Tech Lead / Architect
Standardise one model-access layer for the team, with secrets handling, version pinning, logging and cost tracking, rather than every feature calling providers its own way.
Project / Delivery Manager
Ask for the fallback plan and the cost per request before launch. A feature with no plan for outages or bad output isn't ready.
Scrum Master / Agile
Include "error paths tested" and "evaluation set passing" in the team's definition of done for AI features.
Try it: design the request and the failure plan
- Pick a small feature, such as tagging support tickets with a category and urgency.
- Write the JSON schema with allowed values, the system prompt and the request settings (model version, temperature, max tokens, timeout).
- List four ways the call could fail and exactly what your code does in each case.
Key takeaways
- Call models from your backend; keep keys server-side and add your own auth and limits.
- A request combines a pinned model, a versioned system prompt, messages, limits, temperature and an output format.
- Validate every response like untrusted input, and never execute model output directly.
- Plan for timeouts, rate limits, truncation and outages; test error paths with mocks and quality with an evaluation set.
Key terms
- API
- An interface your software calls to use another service, such as a model provider.
- Exponential backoff
- Retrying after waits that grow each time, often with a little randomness (jitter).
- HTTP 429
- The standard "too many requests" response sent when you exceed a rate limit.
- Stop reason
- Why the model stopped generating: it finished, hit the token limit or requested a tool.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Where should the model provider's API key live?
2. Why pin the model version in production?
3. The model returns JSON with a category not in your allowed list. What should your code do?
4. You receive HTTP 429 from the provider. What is the right response?
5. Which test is fast, free and runs on every commit?