Reference architectures for AI systems
The common patterns for AI systems (assistants, RAG, workflows, agents, batch enrichment and embedded copilots), the shared building blocks of an AI platform, the trade-offs between them, and how to document AI architecture decisions.
๐ฏ By the end of this lesson you will be able to:
- Recognise the main AI system patterns and when each fits
- Describe the shared building blocks of an AI platform
- Weigh trade-offs and record AI architecture decisions clearly
In this lesson
What an AI architect decides
An AI technical architect turns business ideas into systems that are reliable, secure, affordable and possible to evolve. Most of the job isn't choosing a clever model. It is choosing the right pattern, designing the shared building blocks many features reuse, and making the trade-offs explicit, so the organisation doesn't end up with twenty AI features each built differently, with twenty sets of keys, logs and risks.
Six common patterns
| Pattern | Shape | Fits when | Watch out for |
|---|---|---|---|
| Assistant | Chat UI โ model, with instructions | Open-ended help, drafting, Q&A on general knowledge | Hallucination, scope creep |
| RAG | Question โ retrieve โ model answers with sources | Answers must come from your documents | Retrieval quality, permissions, freshness |
| Workflow (pipeline) | Fixed steps, model used for some steps | Known process: classify, extract, summarise | Brittle steps if inputs vary a lot |
| Agent | Model chooses tools in a loop | Steps depend on what's found; flexible tasks | Cost, unpredictability, security |
| Batch enrichment | Offline jobs over many records | Tag, translate or summarise large volumes | Throughput, cost, quality sampling |
| Embedded copilot | AI inside an existing product screen | Help users in context, such as drafting in a form | UX, latency, clear human control |
The shared building blocks
- AI gateway: a single path to all model providers, handling keys, routing, rate limits, caching, fallbacks and logging (Lesson 27).
- Prompt and configuration registry: versioned prompts, model choices and settings, changeable without redeploying.
- Retrieval layer: ingestion, chunking, embeddings, vector and keyword search, with permission filtering.
- Tool layer: approved tools and MCP servers with scoped credentials and approval rules.
- Guardrails: input and output checks, policy enforcement and data protection.
- Observability and evaluation: traces, metrics, cost tracking, evaluation runs and dashboards.
- Data stores: conversation history, memory and feedback, with retention rules.
Key trade-offs
| Trade-off | One side | Other side |
|---|---|---|
| Flexibility vs predictability | Agents adapt to new situations | Workflows are cheaper, faster and easier to test |
| Quality vs cost and speed | Larger models, more context, more steps | Smaller models, focused context, caching |
| Build vs buy | Custom features tuned to your process | Vendor products: faster to adopt, less control |
| Central platform vs team autonomy | Consistency, security, shared cost | Speed for individual teams |
| Real-time vs batch | Immediate answers for users | Lower cost and higher throughput offline |
Non-functional requirements for AI
- Quality targets defined by evaluation thresholds (Lesson 19), not just uptime.
- Latency: time to first token and total time, per use case.
- Cost: per request, per user and per month, with budgets.
- Security and privacy: data classification, residency, retention and access control (Lesson 26).
- Resilience: behaviour during provider outages, rate limits and model deprecations.
- Auditability: which model, prompt and data produced each output.
Recording decisions
An AI architecture decision record (ADR)
Title: Use RAG with hybrid search for the HR policy assistant Context: 400 policy documents updated monthly; answers must cite sources; employees see only their country's policies Decision: RAG over the policy repository, hybrid search with reranking, permission filter by country, answers must cite sources, refuse when nothing relevant is retrieved Alternatives: fine-tuning (rejected: knowledge changes monthly, no citations); agent (rejected: no actions needed) Consequences: content owners must keep documents current; nightly re-index; retrieval evaluation added to CI
Short, explicit and reviewable. When someone asks "why not fine-tune?" in a year, the answer is written down.
What this means for your role
Developer
Build features on the shared platform blocks instead of calling providers directly, and raise gaps in the platform rather than working around them.
Tester / QA
Ask for the pattern and non-functional requirements of each AI feature; they tell you what to evaluate, from retrieval quality to agent trajectories.
Business Analyst / PO
Match business needs to patterns early: does the user need answers from documents (RAG), a fixed process (workflow) or flexible action (agent)?
Tech Lead / Architect
Own the reference architecture, the shared building blocks and the ADRs. Review new AI features against them before they are built.
Project / Delivery Manager
Expect platform work, such as a gateway and observability, to be planned and funded once, then reused by many features.
Scrum Master / Agile
Help platform and feature teams coordinate: shared components need clear ownership, roadmaps and support channels.
Try it: architect one AI feature
- Pick an AI idea from your organisation and choose its pattern or combination of patterns.
- Sketch which shared building blocks it uses, and list its non-functional requirements with numbers.
- Write a five-line ADR: context, decision, two alternatives rejected and the consequences.
Key takeaways
- Most AI systems use six patterns: assistant, RAG, workflow, agent, batch enrichment and embedded copilot, often combined.
- Shared building blocks (gateway, prompt registry, retrieval, tools, guardrails, observability) prevent every team reinventing them.
- Make trade-offs explicit: flexibility vs predictability, quality vs cost, build vs buy, central vs autonomous.
- Define AI non-functional requirements and record decisions in ADRs.
Key terms
- Reference architecture
- A reusable template of components and patterns for building similar systems.
- AI gateway
- A shared service that all AI traffic passes through for keys, routing, limits, caching and logging.
- ADR
- Architecture decision record: a short document capturing a decision, its context, alternatives and consequences.
- Non-functional requirement
- A requirement about how a system performs, such as speed, cost, security or reliability.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Answers must come from frequently changing internal documents, with citations. Which pattern fits best?
2. Why build a shared AI gateway?
3. A known three-step process runs at high volume. Which pattern is usually best?
4. Which is a non-functional requirement for an AI feature?
5. What belongs in an ADR?