Retrieval-augmented generation (RAG)
How to make AI answer from your own documents, with sources: the two halves of a RAG system, what makes it good or bad, how it compares with fine-tuning, and the questions to ask before building one.
🎯 By the end of this lesson you will be able to:
- Describe the RAG pipeline: prepare documents, retrieve, augment the prompt and generate
- Explain why RAG reduces hallucinations and keeps answers up to date
- Diagnose the common reasons a RAG assistant gives poor answers
In this lesson
Why RAG exists
A language model on its own has three problems for business use. It does not know your private information, such as policies, product docs or past tickets. Its knowledge stops at its training cut-off. And when it does not know, it may invent a plausible answer. Retrieval-augmented generation (RAG) tackles all three by finding relevant passages from your own documents and giving them to the model as part of the prompt, with an instruction to answer only from them.
The RAG pipeline
A RAG system has two halves. Preparation happens ahead of time: documents are collected, split into passages (called chunks), turned into embeddings (Lesson 8) and stored. Answering happens for every question:
- Retrieve: create an embedding for the question and fetch the few most relevant chunks, often with hybrid search.
- Augment: put those chunks into the prompt with clear instructions, for example "Answer only from the sources below. If the answer is not there, say you don't know. Cite the source for each fact."
- Generate: the model writes an answer grounded in the retrieved text, with citations the user can click to check.
An HR policy assistant
Question: How many days can I carry over to next year? Retrieved: Leave Policy v4, section 3.2: "Employees may carry over up to five unused days, to be used by 31 March."
You can carry over up to five unused days, and they must be used by 31 March. [Source: Leave Policy v4, section 3.2]
Without RAG, a model might confidently state a typical number from the internet that is wrong for your company.
What makes RAG good or bad
Most RAG problems are retrieval problems: if the right passage is not retrieved, even the best model cannot answer correctly. Teams check each stage separately.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Says "I don't know" but the answer exists | Right chunk not retrieved | Better chunking, hybrid search, more results, rewriting the question |
| Answer mixes up two policies | Chunks lack context such as document name or version | Add titles and metadata to each chunk; filter by version |
| Quotes an out-of-date rule | Old documents still indexed | Remove or flag old versions; refresh on change |
| Answer adds facts not in the sources | Weak instructions or long, noisy context | Stricter instructions, fewer but better chunks, check citations |
| Shows a document the user shouldn't see | Retrieval ignores permissions | Filter results by the user's access rights before generating |
RAG or fine-tuning?
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model can look up for each question | The model's behaviour, style or skills |
| Best for | Facts that change: policies, docs, products, tickets | Consistent format, tone or a specialised task |
| Updating knowledge | Update the documents, done | Retrain with new examples |
| Shows sources | Yes, naturally | No |
For "answer questions about our information", RAG is almost always the right first step. Fine-tuning is for teaching behaviour, and the two can be combined.
Questions to ask before building a RAG assistant
- Which documents are in scope, who owns them, and are they accurate and current?
- Who may see which documents, and how will the system enforce that?
- What are 30 to 50 real questions users ask, with the correct answers and sources? This becomes the test set (Lesson 11).
- What should happen when the answer is not in the documents?
- How will documents be kept up to date in the system?
What this means for your role
Developer
Most of your effort goes into ingestion and retrieval: chunking, metadata, hybrid search, permission filters and refresh jobs. Log which chunks were used for every answer so problems can be traced.
Tester / QA
Test retrieval and generation separately: for each test question, was the right source retrieved, and was the answer faithful to it? Include questions with no answer in the documents.
Business Analyst / PO
Define scope, sources of truth, document owners and the expected behaviour when no answer exists. Collect the real questions and correct answers early.
Tech Lead / Architect
Design for permissions, freshness and traceability from day one. Plan how documents flow in, how they are versioned and how they are removed.
Project / Delivery Manager
Content clean-up is often the biggest hidden task. Budget time for owners to fix, retire and tag documents before launch.
Scrum Master / Agile
A team knowledge assistant over your own wiki is a great first RAG pilot. Track the questions it fails on as backlog items to improve the docs.
Try it: a mini RAG by hand
- Choose a public policy or product document of two to three pages.
- Paste it into an AI assistant between <document> tags with the instruction: "Answer only from the document. Quote the sentence you used. If it isn't there, say so."
- Ask three questions: one clearly answered, one partly answered and one not covered. Check whether the quotes are real and whether it admits what it doesn't know.
Key takeaways
- RAG retrieves relevant passages from your documents and makes the model answer from them, with sources.
- It reduces hallucinations, adds private knowledge and stays current when documents are updated.
- Most RAG failures are retrieval failures: chunking, search, metadata and freshness matter most.
- Respect permissions, plan for "no answer" cases, and choose RAG for knowledge and fine-tuning for behaviour.
Key terms
- RAG
- Retrieval-augmented generation: answering with a model after retrieving relevant passages from your own data.
- Chunk
- A passage of a document stored and retrieved as one unit.
- Grounding
- Making a model base its answer on provided sources rather than memory.
- Fine-tuning
- Further training a model on examples to change its behaviour or skills.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. What does RAG add to a language model?
2. A RAG assistant says "I don't know" although the answer is in the documents. What is the most likely cause?
3. Your company wants an assistant that answers questions about policies that change monthly. What is the best first approach?
4. Why must retrieval respect user permissions?
5. Which instruction best reduces made-up answers in RAG?