Nishaglobal Education logo

Your first LLM-powered feature

The architecture of a real AI feature, what goes into a model API call, getting validated structured output, handling errors, timeouts and rate limits, keeping keys safe, and testing it properly.

Lesson 14 of 4418 min readQuiz: 5 questions

๐ŸŽฏ By the end of this lesson you will be able to:

  • Sketch the architecture of an AI feature and explain why the model is called from your backend
  • Describe the parts of a model API request and response
  • Build in validation, error handling, streaming and tests from the start
In this lesson
  1. The architecture of a simple AI feature
  2. What goes into a model API call
  3. Validate everything that comes back
  4. Errors, timeouts and rate limits
  5. User experience: streaming and honesty
  6. Testing an AI feature
  7. What this means for your role
  8. Key takeaways
  9. Quiz

The architecture of a simple AI feature

Most AI features follow the same shape. The user interacts with your app. Your backend builds the prompt from a template, the user's input and any data it needs, then calls the model provider's API. It checks the response, stores what it needs for monitoring, and returns a result to the user. The model is a service you call, much like a payment or email API.

User's appbrowser / mobileYour backendAuth + limitsPrompt template v3Validate outputLog + cost tracking๐Ÿ”‘ API key stays hererequestresponseModel APIprovider
The browser never talks to the model provider directly. Your backend owns the key, the prompt, validation and logging.

What goes into a model API call

Provider APIs differ in detail, but almost all chat-style calls contain the same parts. The example below is simplified pseudo-code, not any specific SDK.

A typical request (pseudo-code)

response = model_api.create({
  model: "<model name, pinned to a version>",
  system: load_template("ticket-triage", version="v3"),
  messages: [{ role: "user", content: ticket_text }],
  max_output_tokens: 300,
  temperature: 0,
  response_format: TRIAGE_JSON_SCHEMA,
  timeout_seconds: 20
})

The response contains the generated content (here, JSON), token counts for input and output, a stop reason (finished, hit the token limit, or asked to call a tool) and an ID you should log.

PartWhy it matters
Model and versionPin a version so behaviour doesn't change under you; upgrade deliberately after re-running evals
System promptStable instructions loaded from a versioned template
MessagesThe user input and any retrieved context, clearly separated
Max output tokensCaps cost and latency; check the stop reason for truncated answers
TemperatureLow for extraction and classification, higher for creative text
Response format / schemaAsks for, or enforces, structured output
TimeoutA slow call shouldn't hang your app

Validate everything that comes back

Treat model output like input from an untrusted user. Parse it, validate it against your schema, check allowed values and lengths, and decide what to do when it fails: retry once with the error message, fall back to a safe default, or send the case to a person.

Validation flow (pseudo-code)

result = parse_json(response.content)
if not TRIAGE_SCHEMA.is_valid(result):
    result = retry_once_with_feedback(errors)
if not TRIAGE_SCHEMA.is_valid(result):
    return route_to_human(ticket)
if result.category not in ALLOWED_CATEGORIES:
    result.category = "other"

The feature keeps working even when the model misbehaves, and bad outputs never reach your database unchecked.

Errors, timeouts and rate limits

  • Timeouts: set them on every call, and show the user a helpful message if one fires.
  • Rate limits: providers return a "too many requests" error (HTTP 429). Retry with exponential backoff and jitter, and queue non-urgent work.
  • Transient errors: retry a small number of times; don't retry errors caused by a bad request.
  • Truncation: if the stop reason says the token limit was hit, the answer may be cut off. Handle it rather than showing half a sentence.
  • Fallbacks: a secondary model, a cached answer or a simple non-AI path keeps the feature usable during an outage.

User experience: streaming and honesty

For text users read, stream the response so words appear immediately. Show that the content is AI-generated where users rely on it, offer a way to report problems, and make it easy to edit or undo AI suggestions. For structured tasks, such as tagging or routing, streaming is unnecessary. Return the validated result instead.

Testing an AI feature

Test typeWhat it checksRuns
Unit tests with a mocked modelPrompt building, parsing, validation, retries, fallbacksEvery commit, fast and free
Evaluation set (Lesson 11)Real quality on realistic inputs with the real modelOn prompt, model or retrieval changes
Contract testsYour code still handles the provider's response formatRegularly and on SDK upgrades
Load and cost testsLatency, rate limits and cost at expected volumeBefore launch and major changes

What this means for your role

Developer

Build a thin, well-tested wrapper around model calls: versioned prompts, schema validation, retries with backoff, timeouts, streaming and structured logging of model, prompt version, tokens and latency.

Tester / QA

Mock the model to test error paths: invalid JSON, wrong values, timeouts, 429s and truncated output. Then run the evaluation set against the real model for quality.

Business Analyst / PO

Specify what happens when the AI fails or is unsure: the fallback, the message users see, and when a person takes over.

Tech Lead / Architect

Standardise one model-access layer for the team, with secrets handling, version pinning, logging and cost tracking, rather than every feature calling providers its own way.

Project / Delivery Manager

Ask for the fallback plan and the cost per request before launch. A feature with no plan for outages or bad output isn't ready.

Scrum Master / Agile

Include "error paths tested" and "evaluation set passing" in the team's definition of done for AI features.

Try it: design the request and the failure plan

  1. Pick a small feature, such as tagging support tickets with a category and urgency.
  2. Write the JSON schema with allowed values, the system prompt and the request settings (model version, temperature, max tokens, timeout).
  3. List four ways the call could fail and exactly what your code does in each case.

Key takeaways

  • Call models from your backend; keep keys server-side and add your own auth and limits.
  • A request combines a pinned model, a versioned system prompt, messages, limits, temperature and an output format.
  • Validate every response like untrusted input, and never execute model output directly.
  • Plan for timeouts, rate limits, truncation and outages; test error paths with mocks and quality with an evaluation set.

Key terms

API
An interface your software calls to use another service, such as a model provider.
Exponential backoff
Retrying after waits that grow each time, often with a little randomness (jitter).
HTTP 429
The standard "too many requests" response sent when you exceed a rate limit.
Stop reason
Why the model stopped generating: it finished, hit the token limit or requested a tool.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. Where should the model provider's API key live?

  2. 2. Why pin the model version in production?

  3. 3. The model returns JSON with a category not in your allowed list. What should your code do?

  4. 4. You receive HTTP 429 from the provider. What is the right response?

  5. 5. Which test is fast, free and runs on every commit?

Answer all 5 questions to check.

Practise more