Nishaglobal Education logo

RAG in practice

Building a RAG system that works on real documents: chunking strategies, metadata, choosing embeddings and a vector store, hybrid search and reranking, citations, permission filtering, keeping the index fresh and measuring retrieval quality.

Lesson 15 of 4419 min readQuiz: 5 questions

🎯 By the end of this lesson you will be able to:

  • Choose a chunking strategy and attach useful metadata to every chunk
  • Improve retrieval with hybrid search, filters and reranking
  • Measure retrieval and answer quality separately, and keep the index fresh
In this lesson
  1. From demo to real documents
  2. Chunking: how you split documents
  3. Metadata: the context a chunk carries
  4. Better retrieval
  5. Answering with citations
  6. Permissions and freshness
  7. Measuring RAG quality
  8. What this means for your role
  9. Key takeaways
  10. Quiz

From demo to real documents

A RAG demo on ten clean documents is easy. Real collections are messy: PDFs with tables, duplicated pages, old versions, mixed languages and documents only some people may see. Lesson 9 explained the idea. This lesson covers the engineering choices that decide whether a RAG system is trusted or abandoned.

Chunking: how you split documents

Chunks are what retrieval finds and what the model reads. Too large, and one chunk mixes topics and wastes tokens. Too small, and a chunk loses the context needed to understand it.

Leave Policy v4§1 Eligibility … §2 Requesting leave … §3 Carry-over … §4 Sick leave …Chunk 1§1 + start of §2Chunk 2end of §2 + §3Chunk 3§3 + start of §4Chunk 4§4Overlapping edges keep context · each chunk stores metadata:title · section · version · date · access group
Split by structure where you can (headings, sections), keep chunks a moderate size, and overlap neighbours slightly so sentences at the edges keep their context.
StrategyHow it worksGood for
Fixed size with overlapEvery N tokens, overlapping by a few sentencesPlain text with little structure
By structureSplit at headings, sections, list items or table rowsPolicies, manuals, documentation
Parent–childSearch small chunks, then give the model the surrounding larger sectionLong documents where context matters
Special handlingTables, code and FAQs kept intact or converted carefullyMixed-format content

Metadata: the context a chunk carries

Store useful facts with each chunk: document title, section heading, version, date, owner, language, product and access group. Metadata lets you filter (only current versions, only this product), helps the model understand where text came from, and makes citations meaningful. Many teams also prepend the document title and heading to the chunk text before embedding it.

Better retrieval

  • Embeddings: use a well-supported embedding model that handles your languages. Always embed queries and chunks with the same model, and re-embed everything if you change it.
  • Vector store: options range from a vector extension in a database you already run to a managed vector database. Choose based on scale, filtering needs and what your team can operate.
  • Hybrid search: combine keyword and semantic results so product codes and paraphrases both work.
  • Filters before ranking: apply permissions, version and product filters as part of the search, not afterwards.
  • Reranking: retrieve, say, 30 candidates quickly, then use a reranker model to put the best few on top. This often improves quality noticeably.
  • Query rewriting: have the model turn a vague or conversational question into a clear search query, especially in multi-turn chats.

Answering with citations

Number the retrieved chunks in the prompt, require the model to cite them, and show citations as links to the source passage. Then check citations automatically: every cited number must exist, and spot-checks should confirm the quoted text supports the claim. If nothing relevant was retrieved, return a clear "I couldn't find this in our documents" with a way to ask a person, rather than letting the model guess.

Permissions and freshness

  • Mirror source permissions: store each chunk's access groups and filter by the current user's rights on every query.
  • Sync changes: re-index changed documents, remove deleted ones and retire old versions quickly. Stale chunks are a top cause of wrong answers.
  • Track lineage: know which source file and version each chunk came from, so you can fix problems at the source.

Measuring RAG quality

StageQuestionExample measure
RetrievalDid we fetch the right passages?Share of test questions with the correct source in the top 5 (recall@5)
FaithfulnessIs the answer supported by the passages?Share of claims backed by a cited chunk
Answer qualityIs it correct, complete and clear?Rubric score (Lesson 11)
RefusalsDoes it admit when the answer isn't there?Correct "not found" rate on no-answer questions

What this means for your role

Developer

Build the ingestion pipeline as carefully as the chat UI: parsing, chunking, metadata, embeddings, permission tags and incremental re-indexing. Log retrieved chunk IDs with every answer.

Tester / QA

Own the RAG evaluation set: questions with known source passages, measured for retrieval and faithfulness separately, plus no-answer and permission tests.

Business Analyst / PO

Identify the sources of truth, their owners, update frequency and access rules. Poor or conflicting source documents are a business problem to solve before launch.

Tech Lead / Architect

Choose the vector store and search approach the team can operate, design permission filtering and lineage, and set a re-embedding strategy for model changes.

Project / Delivery Manager

Plan for content work and re-indexing operations, not just the build. Track retrieval and faithfulness scores as launch criteria.

Scrum Master / Agile

Turn "couldn't find it" answers into a visible backlog of missing or unclear documentation for the team to fix.

Try it: plan a RAG index for your team's docs

  1. List the document sources you would include and who owns each.
  2. Choose a chunking strategy for each source type and the metadata fields you would store.
  3. Write ten test questions, each with the document and section that should be retrieved. That is your first retrieval evaluation set.

Key takeaways

  • Chunk by structure where possible, at a moderate size with slight overlap, and test settings on your own questions.
  • Rich metadata enables filtering, better context and meaningful citations.
  • Hybrid search, filters, reranking and query rewriting improve retrieval more than prompt tweaks.
  • Enforce permissions, keep the index fresh, and measure retrieval and faithfulness separately.

Key terms

Chunk overlap
Repeating a little text between neighbouring chunks so edge sentences keep context.
Reranker
A model that reorders retrieved candidates by relevance to the question.
Recall@k
The share of test questions where a correct source appears in the top k results.
Faithfulness
How fully an answer is supported by the retrieved sources.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. Why add a little overlap between neighbouring chunks?

  2. 2. Where should permission filters be applied?

  3. 3. What does a reranker do?

  4. 4. An answer is wrong. What should you check first?

  5. 5. You switch to a new embedding model. What must you do?

Answer all 5 questions to check.

Practise more