Nishaglobal Education logo

Reference architectures for AI systems

The common patterns for AI systems (assistants, RAG, workflows, agents, batch enrichment and embedded copilots), the shared building blocks of an AI platform, the trade-offs between them, and how to document AI architecture decisions.

Lesson 24 of 4418 min readQuiz: 5 questions

๐ŸŽฏ By the end of this lesson you will be able to:

  • Recognise the main AI system patterns and when each fits
  • Describe the shared building blocks of an AI platform
  • Weigh trade-offs and record AI architecture decisions clearly
In this lesson
  1. What an AI architect decides
  2. Six common patterns
  3. The shared building blocks
  4. Key trade-offs
  5. Non-functional requirements for AI
  6. Recording decisions
  7. What this means for your role
  8. Key takeaways
  9. Quiz

What an AI architect decides

An AI technical architect turns business ideas into systems that are reliable, secure, affordable and possible to evolve. Most of the job isn't choosing a clever model. It is choosing the right pattern, designing the shared building blocks many features reuse, and making the trade-offs explicit, so the organisation doesn't end up with twenty AI features each built differently, with twenty sets of keys, logs and risks.

Six common patterns

PatternShapeFits whenWatch out for
AssistantChat UI โ†’ model, with instructionsOpen-ended help, drafting, Q&A on general knowledgeHallucination, scope creep
RAGQuestion โ†’ retrieve โ†’ model answers with sourcesAnswers must come from your documentsRetrieval quality, permissions, freshness
Workflow (pipeline)Fixed steps, model used for some stepsKnown process: classify, extract, summariseBrittle steps if inputs vary a lot
AgentModel chooses tools in a loopSteps depend on what's found; flexible tasksCost, unpredictability, security
Batch enrichmentOffline jobs over many recordsTag, translate or summarise large volumesThroughput, cost, quality sampling
Embedded copilotAI inside an existing product screenHelp users in context, such as drafting in a formUX, latency, clear human control

The shared building blocks

Support copilotPolicy assistantDoc summariserSales draftsfeatures (owned by product teams)Prompt registryversioned configRetrievalsearch + permissionsTools / MCPscoped credentialsGuardrailsinput + output checksAI gatewaykeys ยท routing ยท limits ยท caching ยท fallbacks ยท logsData storeshistory + retentionObservability + evaluation across every layerProvider AProvider BSelf-hosted
Features sit on shared platform services: a gateway to models, prompt and configuration management, retrieval, tools, guardrails and observability with evaluation.
  • AI gateway: a single path to all model providers, handling keys, routing, rate limits, caching, fallbacks and logging (Lesson 27).
  • Prompt and configuration registry: versioned prompts, model choices and settings, changeable without redeploying.
  • Retrieval layer: ingestion, chunking, embeddings, vector and keyword search, with permission filtering.
  • Tool layer: approved tools and MCP servers with scoped credentials and approval rules.
  • Guardrails: input and output checks, policy enforcement and data protection.
  • Observability and evaluation: traces, metrics, cost tracking, evaluation runs and dashboards.
  • Data stores: conversation history, memory and feedback, with retention rules.

Key trade-offs

Trade-offOne sideOther side
Flexibility vs predictabilityAgents adapt to new situationsWorkflows are cheaper, faster and easier to test
Quality vs cost and speedLarger models, more context, more stepsSmaller models, focused context, caching
Build vs buyCustom features tuned to your processVendor products: faster to adopt, less control
Central platform vs team autonomyConsistency, security, shared costSpeed for individual teams
Real-time vs batchImmediate answers for usersLower cost and higher throughput offline

Non-functional requirements for AI

  • Quality targets defined by evaluation thresholds (Lesson 19), not just uptime.
  • Latency: time to first token and total time, per use case.
  • Cost: per request, per user and per month, with budgets.
  • Security and privacy: data classification, residency, retention and access control (Lesson 26).
  • Resilience: behaviour during provider outages, rate limits and model deprecations.
  • Auditability: which model, prompt and data produced each output.

Recording decisions

An AI architecture decision record (ADR)

Title: Use RAG with hybrid search for the HR policy assistant
Context: 400 policy documents updated monthly; answers must cite sources; employees see only their country's policies
Decision: RAG over the policy repository, hybrid search with reranking, permission filter by country, answers must cite sources, refuse when nothing relevant is retrieved
Alternatives: fine-tuning (rejected: knowledge changes monthly, no citations); agent (rejected: no actions needed)
Consequences: content owners must keep documents current; nightly re-index; retrieval evaluation added to CI

Short, explicit and reviewable. When someone asks "why not fine-tune?" in a year, the answer is written down.

What this means for your role

Developer

Build features on the shared platform blocks instead of calling providers directly, and raise gaps in the platform rather than working around them.

Tester / QA

Ask for the pattern and non-functional requirements of each AI feature; they tell you what to evaluate, from retrieval quality to agent trajectories.

Business Analyst / PO

Match business needs to patterns early: does the user need answers from documents (RAG), a fixed process (workflow) or flexible action (agent)?

Tech Lead / Architect

Own the reference architecture, the shared building blocks and the ADRs. Review new AI features against them before they are built.

Project / Delivery Manager

Expect platform work, such as a gateway and observability, to be planned and funded once, then reused by many features.

Scrum Master / Agile

Help platform and feature teams coordinate: shared components need clear ownership, roadmaps and support channels.

Try it: architect one AI feature

  1. Pick an AI idea from your organisation and choose its pattern or combination of patterns.
  2. Sketch which shared building blocks it uses, and list its non-functional requirements with numbers.
  3. Write a five-line ADR: context, decision, two alternatives rejected and the consequences.

Key takeaways

  • Most AI systems use six patterns: assistant, RAG, workflow, agent, batch enrichment and embedded copilot, often combined.
  • Shared building blocks (gateway, prompt registry, retrieval, tools, guardrails, observability) prevent every team reinventing them.
  • Make trade-offs explicit: flexibility vs predictability, quality vs cost, build vs buy, central vs autonomous.
  • Define AI non-functional requirements and record decisions in ADRs.

Key terms

Reference architecture
A reusable template of components and patterns for building similar systems.
AI gateway
A shared service that all AI traffic passes through for keys, routing, limits, caching and logging.
ADR
Architecture decision record: a short document capturing a decision, its context, alternatives and consequences.
Non-functional requirement
A requirement about how a system performs, such as speed, cost, security or reliability.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. Answers must come from frequently changing internal documents, with citations. Which pattern fits best?

  2. 2. Why build a shared AI gateway?

  3. 3. A known three-step process runs at high volume. Which pattern is usually best?

  4. 4. Which is a non-functional requirement for an AI feature?

  5. 5. What belongs in an ADR?

Answer all 5 questions to check.

Practise more