Retrieval-Augmented Generation (RAG) Explained: How It Works, Design Choices, Evaluation and When to Use It

A practical guide to retrieval-augmented generation: how RAG grounds language models in your documents, chunking, hybrid search and re-ranking, agentic and graph-based variants, how to evaluate faithfulness, security and access control, and how RAG compares with long context windows and fine-tuning.

PUBLISHED // UPDATED
DURATION // 06 MIN READ
Retrieval-Augmented Generation (RAG) Explained: How It Works, Design Choices, Evaluation and When to Use It
FIG.01 // VISUAL DOCUMENTATION Retrieval-Augmented Generation (RAG) Explained: How It Works, Design Choices, Evaluation and When to Use It

Retrieval-augmented generation (RAG) is the most common way to make a language model answer questions using your own information, such as product manuals, policies, contracts or research notes. Instead of relying only on what the model absorbed in training, a RAG system searches a knowledge base for relevant passages and gives them to the model alongside the question. It powers many customer-support assistants, internal knowledge tools and research products that cite their sources. This guide explains how it works, the design decisions that matter most, how to evaluate it and when other approaches fit better.

How RAG works

A basic RAG pipeline has two phases.

Indexing (done in advance). Documents are split into chunks, each chunk is converted into an embedding, a list of numbers that captures its meaning, and the embeddings are stored in a vector database or search index, often alongside a traditional keyword index.

Answering (at query time). The user’s question is embedded the same way, the system retrieves the most similar chunks, and those chunks are placed in the model’s prompt with instructions to answer from them and cite them. The model then writes an answer grounded in the retrieved text.

StageMain choicesCommon mistakes
IngestionFile parsing, cleaning, metadataLosing tables, headings and page structure
ChunkingSize, overlap, splitting by sectionChunks too small to make sense or too big to be precise
RetrievalEmbeddings, keyword search, filtersRelying on embeddings alone for codes and names
Re-rankingCross-encoder models on top candidatesSkipping it and sending noisy context
GenerationPrompt, citation format, refusal rulesLetting the model answer beyond the sources

Why organisations use RAG

Grounding. Language models can produce plausible but false answers when they lack information. Giving them relevant source text, and instructing them to say when the sources do not contain an answer, reduces fabrication, though it does not eliminate it. Models can still misread or over-generalise from what they retrieve.

Fresh knowledge. Models are frozen at their training cutoff. With RAG, new documents become available as soon as they are indexed, without retraining.

Citations. Returning the passages behind each answer lets users check claims, which builds trust and makes errors easier to spot.

Access control. Retrieval can respect existing permissions so that each user only sees answers drawn from documents they are allowed to read, something that is very hard to achieve by training knowledge into a model.

Cost. Updating an index is usually far cheaper than fine-tuning a model every time information changes.

How RAG has evolved

Hybrid retrieval. Embedding search captures meaning but can miss exact terms such as product codes, error messages and names. Combining it with keyword search such as BM25, plus metadata filters for dates, products or departments, is now standard.

Re-ranking and context selection. Retrieving a broad set of candidates and then re-ranking them with a more precise model helps place the most relevant passages first. Sending fewer, better passages often beats stuffing the prompt, because models can overlook information buried in long inputs.

Agentic RAG. Instead of a single search, an agent plans several retrieval steps, reformulates queries, looks up related documents and decides when it has enough information. A question comparing quarterly results might trigger separate searches for each quarter and for relevant policy changes. This improves complex answers at the cost of latency and more opportunities for error.

Graph-based RAG. Microsoft researchers proposed GraphRAG, which builds a knowledge graph of entities and relationships from the documents and summarises communities of related information. It targets broad questions, such as the main themes across a large collection, that chunk retrieval answers poorly.

Limitations

RAG cannot answer what the knowledge base does not contain, and it inherits the quality of the documents: outdated, contradictory or badly formatted sources produce poor answers. Retrieval can fail silently, returning plausible but wrong passages, and the model may then answer confidently. RAG also adds latency and engineering complexity compared with calling a model directly, and chunking and parsing still need tuning for each document type.

Security and privacy

RAG systems bring their own risks. Documents can contain hidden instructions, a form of indirect prompt injection, that try to make the model ignore its rules or leak information. Permission-aware retrieval is essential so the assistant does not surface documents a user should not see. Sensitive data in the index should be classified and, where appropriate, excluded or redacted, and logs of queries and answers need the same protection as the documents themselves.

Where RAG came from

The term comes from Lewis et al., 2020, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (arXiv:2005.11401), which combined a dense passage retriever with a sequence-to-sequence generator and showed gains on open-domain question answering. Related work such as REALM (arXiv:2002.08909) and RETRO (arXiv:2112.04426) explored retrieval during pre-training. Today’s production RAG systems are usually simpler: they retrieve chunks from a vector or hybrid index at query time and paste them into the prompt of a general-purpose LLM.

Design decisions that matter most

  • Chunking: Chunk size and overlap strongly affect retrieval quality. Chunks that are too small lose context; chunks that are too large dilute relevance and waste tokens.
  • Hybrid retrieval: Combining keyword search (such as BM25) with embedding search typically helps with exact terms like product codes and names that embeddings handle poorly.
  • Re-ranking: A cross-encoder re-ranker applied to the top candidates often improves precision more cheaply than switching embedding models.
  • Parsing: Extracting tables, headings and page structure correctly from PDFs and slides often matters more than the choice of embedding model.
  • Instructions: Tell the model to answer only from the provided sources, cite them and say when the answer is not there.

How to evaluate a RAG system

Evaluate retrieval and generation separately. For retrieval, measure whether the correct passage appears in the top results (recall@k). For generation, check faithfulness (is the answer supported by the retrieved text?) and answer relevance. Open-source tools such as Ragas implement these metrics, but a small, hand-labeled set of real user questions remains the most reliable test.

RAG vs. long context vs. fine-tuning

  • Long context windows can replace RAG for small, static document sets, but cost grows with every token sent, and models can still miss information buried in the middle of long inputs (Liu et al., “Lost in the Middle”).
  • Fine-tuning changes style and behavior well but is a poor way to inject frequently changing facts.
  • RAG remains the default when knowledge changes often, must be access-controlled, or needs citations.

Our analysis

RAG has become standard infrastructure for applying language models to an organisation’s own information because it offers freshness, citations and access control that training alone cannot. Its quality depends less on the model than on the unglamorous parts: document parsing, chunking, hybrid retrieval, re-ranking and careful instructions. Teams should evaluate retrieval and generation separately on real questions, design for permissions and prompt-injection risks from the start, and consider long context or fine-tuning only where they fit the problem better.