RAG Explained: How Retrieval-Augmented Generation Actually Works

Key takeaways

1. RAG is a retrieval problem before it is a model problem: a perfect model with irrelevant retrieved context still produces a wrong answer.
2. Chunking strategy matters more than most teams expect: chunks that are too large dilute relevance; chunks that are too small lose context.
3. Embeddings capture semantic similarity, not truth: a highly similar chunk can still be outdated or wrong.
4. Grounding requires the model to cite what it retrieved: without an explicit instruction to answer only from context, models still blend in memorized knowledge.
5. Source freshness and access control are production concerns, not implementation details — an ungoverned RAG pipeline can leak access-restricted documents or serve stale answers with confidence.

What does the RAG pipeline actually look like?

Five stages, in order: ingestion, chunking, embedding, storage, and retrieval-plus-generation.

1. Ingestion: pull documents from their source — a knowledge base, a database, a file store — and normalize them to plain text.
2. Chunking: split each document into smaller passages, typically a few hundred tokens, often with some overlap between adjacent chunks so a fact split across a boundary isn't lost entirely.
3. Embedding: convert each chunk into a numeric vector using an embedding model. Chunks with similar meaning end up close together in that vector space, regardless of exact wording.
4. Storage: store the vectors and their source chunks in a vector database or a Postgres extension like pgvector, indexed for fast similarity search.
5. Retrieval and generation: at query time, embed the user's question, find the nearest chunks by vector similarity, and pass those chunks plus the question to the language model as context for its answer.

Why does RAG still hallucinate?

Adding retrieval does not remove a model's tendency to generate plausible-sounding text — it changes what the model has available to ground its answer in. Three common failure modes:

A RAG system that returns a confident answer with no indication of source or freshness is not more trustworthy than a model with no retrieval at all — it just sounds more trustworthy.

What does "grounding" mean in practice?

Grounding means the model's answer is verifiably derived from the retrieved context, not just plausible given it. In practice this means three things in the system prompt and pipeline design: instruct the model to answer strictly from the provided passages, require it to say when the context doesn't contain an answer instead of filling the gap from memory, and attach citations — which chunk, which document, which timestamp — so an answer can be checked against its source.

SYSTEM: Answer only using the CONTEXT below.
If the context does not contain the answer, say so explicitly.
Cite the source document for every claim.

CONTEXT:
[retrieved chunks with source + timestamp metadata]

QUESTION: {user_question}

When is RAG the wrong tool?

RAG solves "the model needs facts it wasn't trained on, or facts that change over time." It does not solve problems better handled by fine-tuning (teaching a model a consistent style or narrow skill), structured lookups (a direct database query is more reliable than embedding-similarity search for exact-match facts like an order status), or reasoning tasks that don't depend on external documents at all. Reaching for RAG by default, even when a simple API call or SQL query would answer the question deterministically, adds latency and a new failure surface for no benefit.