RAG in practice
Building a RAG system that works on real documents: chunking strategies, metadata, choosing embeddings and a vector store, hybrid search and reranking, citations, permission filtering, keeping the index fresh and measuring retrieval quality.
🎯 By the end of this lesson you will be able to:
- Choose a chunking strategy and attach useful metadata to every chunk
- Improve retrieval with hybrid search, filters and reranking
- Measure retrieval and answer quality separately, and keep the index fresh
In this lesson
From demo to real documents
A RAG demo on ten clean documents is easy. Real collections are messy: PDFs with tables, duplicated pages, old versions, mixed languages and documents only some people may see. Lesson 9 explained the idea. This lesson covers the engineering choices that decide whether a RAG system is trusted or abandoned.
Chunking: how you split documents
Chunks are what retrieval finds and what the model reads. Too large, and one chunk mixes topics and wastes tokens. Too small, and a chunk loses the context needed to understand it.
| Strategy | How it works | Good for |
|---|---|---|
| Fixed size with overlap | Every N tokens, overlapping by a few sentences | Plain text with little structure |
| By structure | Split at headings, sections, list items or table rows | Policies, manuals, documentation |
| Parent–child | Search small chunks, then give the model the surrounding larger section | Long documents where context matters |
| Special handling | Tables, code and FAQs kept intact or converted carefully | Mixed-format content |
Metadata: the context a chunk carries
Store useful facts with each chunk: document title, section heading, version, date, owner, language, product and access group. Metadata lets you filter (only current versions, only this product), helps the model understand where text came from, and makes citations meaningful. Many teams also prepend the document title and heading to the chunk text before embedding it.
Better retrieval
- Embeddings: use a well-supported embedding model that handles your languages. Always embed queries and chunks with the same model, and re-embed everything if you change it.
- Vector store: options range from a vector extension in a database you already run to a managed vector database. Choose based on scale, filtering needs and what your team can operate.
- Hybrid search: combine keyword and semantic results so product codes and paraphrases both work.
- Filters before ranking: apply permissions, version and product filters as part of the search, not afterwards.
- Reranking: retrieve, say, 30 candidates quickly, then use a reranker model to put the best few on top. This often improves quality noticeably.
- Query rewriting: have the model turn a vague or conversational question into a clear search query, especially in multi-turn chats.
Answering with citations
Number the retrieved chunks in the prompt, require the model to cite them, and show citations as links to the source passage. Then check citations automatically: every cited number must exist, and spot-checks should confirm the quoted text supports the claim. If nothing relevant was retrieved, return a clear "I couldn't find this in our documents" with a way to ask a person, rather than letting the model guess.
Permissions and freshness
- Mirror source permissions: store each chunk's access groups and filter by the current user's rights on every query.
- Sync changes: re-index changed documents, remove deleted ones and retire old versions quickly. Stale chunks are a top cause of wrong answers.
- Track lineage: know which source file and version each chunk came from, so you can fix problems at the source.
Measuring RAG quality
| Stage | Question | Example measure |
|---|---|---|
| Retrieval | Did we fetch the right passages? | Share of test questions with the correct source in the top 5 (recall@5) |
| Faithfulness | Is the answer supported by the passages? | Share of claims backed by a cited chunk |
| Answer quality | Is it correct, complete and clear? | Rubric score (Lesson 11) |
| Refusals | Does it admit when the answer isn't there? | Correct "not found" rate on no-answer questions |
What this means for your role
Developer
Build the ingestion pipeline as carefully as the chat UI: parsing, chunking, metadata, embeddings, permission tags and incremental re-indexing. Log retrieved chunk IDs with every answer.
Tester / QA
Own the RAG evaluation set: questions with known source passages, measured for retrieval and faithfulness separately, plus no-answer and permission tests.
Business Analyst / PO
Identify the sources of truth, their owners, update frequency and access rules. Poor or conflicting source documents are a business problem to solve before launch.
Tech Lead / Architect
Choose the vector store and search approach the team can operate, design permission filtering and lineage, and set a re-embedding strategy for model changes.
Project / Delivery Manager
Plan for content work and re-indexing operations, not just the build. Track retrieval and faithfulness scores as launch criteria.
Scrum Master / Agile
Turn "couldn't find it" answers into a visible backlog of missing or unclear documentation for the team to fix.
Try it: plan a RAG index for your team's docs
- List the document sources you would include and who owns each.
- Choose a chunking strategy for each source type and the metadata fields you would store.
- Write ten test questions, each with the document and section that should be retrieved. That is your first retrieval evaluation set.
Key takeaways
- Chunk by structure where possible, at a moderate size with slight overlap, and test settings on your own questions.
- Rich metadata enables filtering, better context and meaningful citations.
- Hybrid search, filters, reranking and query rewriting improve retrieval more than prompt tweaks.
- Enforce permissions, keep the index fresh, and measure retrieval and faithfulness separately.
Key terms
- Chunk overlap
- Repeating a little text between neighbouring chunks so edge sentences keep context.
- Reranker
- A model that reorders retrieved candidates by relevance to the question.
- Recall@k
- The share of test questions where a correct source appears in the top k results.
- Faithfulness
- How fully an answer is supported by the retrieved sources.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Why add a little overlap between neighbouring chunks?
2. Where should permission filters be applied?
3. What does a reranker do?
4. An answer is wrong. What should you check first?
5. You switch to a new embedding model. What must you do?