How a RAG answer actually gets built
Retrieval-augmented generation, one stage at a time: chunking, embeddings, hybrid search, reranking and grounded answers. Watch where accuracy is won and lost before the model ever writes a word.
Note: this is an illustrative client-side model — the numbers teach the why, they aren't from a real database.
Step 1 of 7
The model has never seen your documents
A language model knows what was in its training data. It does not know your refund policy, last quarter's numbers, or the runbook your team wrote in March.
Ask anyway and you get fluent, confident, wrong. Retrieval-augmented generation fixes the input rather than the model: find the right passages first, then ask the question with those passages attached.
No retrieval step: the model answers from memory alone.
- Right chunk retrieved
- 0%
- Context tokens sent
- —
- Answer grounded
- No
- Added latency (ms)
- —
Nothing here is checkable. The model cannot cite a source it never saw.
Step 2 of 7
Stage 1: cut documents into pieces that still make sense
Documents are too long to send whole, so they are split into chunks. This is the stage teams rush, and it quietly sets the ceiling on everything downstream.
Split too small and a chunk loses the context that made it meaningful. Split too large and one chunk covers four topics, so matching it to a question gets vague.
# naive: fixed 500 characters, mid-sentence cuts
chunks = [text[i:i+500] for i in range(0, len(text), 500)]
# better: split on structure, keep an overlap, carry the heading
chunks = split_on_headings(doc, max_tokens=400, overlap=50,
prefix=lambda c: c.heading)one document becomes many retrievable chunks
- Right chunk retrieved
- —
- Context tokens sent
- —
- Answer grounded
- —
- Added latency (ms)
- —
Chunking decides what can be found later. No amount of clever search recovers a badly cut corpus.
Step 3 of 7
Stage 2: turn meaning into coordinates
Each chunk is passed through an embedding model and comes back as a list of numbers, a vector. Chunks about similar things land near each other.
That is what lets a question about "cancelling my plan" match a passage titled "ending a subscription" with no shared keywords at all.
vec = embed("how do I end my subscription?")
-- nearest neighbours by cosine distance
SELECT id, text
FROM chunks
ORDER BY embedding <=> :vec
LIMIT 20;What you have learned so far
- Chunk on structure and keep the heading, because a chunk is retrieved alone and must stand alone.
The index is queried with the question, embedded the same way.
- Right chunk retrieved
- 61%
- Context tokens sent
- 8,000
- Answer grounded
- —
- Added latency (ms)
- 40
Good on paraphrase, weak on exact tokens. Search for error code E4021 and semantic similarity will happily return E4012.
Step 4 of 7
Stage 3: run keyword search too, then merge
Keyword search is excellent at exactly what vectors are bad at: identifiers, names, error codes, rare words. So run both and merge the two result lists.
The usual merge is reciprocal rank fusion, which scores a chunk by its position in each list rather than by raw scores, so you never have to make two different scoring scales comparable.
# reciprocal rank fusion over two result lists
def rrf(lists, k=60):
scores = defaultdict(float)
for ranked in lists:
for rank, doc_id in enumerate(ranked):
scores[doc_id] += 1 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
merged = rrf([vector_hits, keyword_hits])What you have learned so far
- Chunk on structure and keep the heading, because a chunk is retrieved alone and must stand alone.
- Embeddings match meaning rather than words, which is why paraphrased questions still find the right passage.
- Right chunk retrieved
- 84%
- Context tokens sent
- 12,000
- Answer grounded
- —
- Added latency (ms)
- 90
Recall is up, but so is the pile. Twenty candidate chunks is far more than the answer needs, and the right one may sit at position 14.
Step 5 of 7
Stage 4: put the genuinely relevant chunk first
Retrieval optimises for speed across millions of chunks, so it is approximate by design. A reranker then reads the question together with each candidate and scores the pair properly.
It is far too slow to run over the whole corpus, which is exactly why it runs over twenty candidates instead.
# retrieve wide, rerank narrow, send few candidates = hybrid_search(q, limit=20) ranked = reranker.score(query=q, docs=candidates) context = ranked[:4] # 4 chunks, not 20
What you have learned so far
- Chunk on structure and keep the heading, because a chunk is retrieved alone and must stand alone.
- Embeddings match meaning rather than words, which is why paraphrased questions still find the right passage.
- Vector and keyword search fail on opposite inputs, so fuse both lists instead of choosing one.
- Right chunk retrieved
- 84%
- Context tokens sent
- 2,600
- Answer grounded
- Likely
- Added latency (ms)
- 320
Better answers and a cheaper prompt, paid for with latency. A reranking pass is the slowest hop in the chain.
Step 6 of 7
Stage 5: make the model cite what it used
The last stage is a prompt discipline, not a model upgrade. Pass the chunks with identifiers, require a citation per claim, and allow the model to say it does not know.
Now an answer is checkable. A reviewer can open the cited chunk and confirm the sentence, which is the difference between a demo and something a support team is allowed to use.
SYSTEM: Answer only from the passages below. Cite the passage id after each claim, like [c7]. If the passages do not contain the answer, say so. [c7] <chunk text> [c2] <chunk text>
What you have learned so far
- Chunk on structure and keep the heading, because a chunk is retrieved alone and must stand alone.
- Embeddings match meaning rather than words, which is why paraphrased questions still find the right passage.
- Vector and keyword search fail on opposite inputs, so fuse both lists instead of choosing one.
- Retrieve wide and rerank narrow, because the cheap search is approximate and the accurate scorer is slow.
Every claim traces back to a passage a human can open.
- Right chunk retrieved
- 84%
- Context tokens sent
- 2,600
- Answer grounded
- Yes
- Added latency (ms)
- 340
Checkable answers. Same model as step one, completely different trustworthiness, because the input changed.
Five stages, and where accuracy is actually won
The model is the last stage and the least of your problems. Retrieval quality decides the answer.
| Stage | Buys you | Costs you | Skip it and |
|---|---|---|---|
| Chunking | Retrievable units | Design effort | Nothing downstream can recover |
| Embeddings | Paraphrase matching | ~40ms | Only exact wording works |
| Hybrid search | Recall 61% → 84% | A second index | Codes and names go missing |
| Reranking | 12,000 → 2,600 tokens | ~230ms | The answer hides at rank 14 |
| Grounding | Checkable claims | Prompt discipline | Confident and unverifiable |
Go deeper
Build a retrieval system that survives real questions
Chunking, evaluation and the unglamorous retrieval work are what separate a demo from something a team can rely on. Crio programs are project based, so you build and measure it yourself.
Run this simulator free
Sign in once to unlock every simulator. Takes 20 seconds.
Questions people ask about this
What chunk size should I use?+
There is no universal number, which is why splitting on document structure beats splitting on a character count. Start by keeping each heading with its section, cap chunks at a few hundred tokens, and add a small overlap so a sentence split across a boundary still appears whole in one chunk. Then measure recall on real questions rather than guessing.
Do I still need keyword search if I have embeddings?+
Usually yes. Vector search matches meaning, so it handles paraphrasing well but is unreliable on exact strings such as error codes, SKUs, version numbers and surnames. Running both and fusing the ranked lists is the standard fix and is typically the largest single accuracy win in a weak system.
What is reciprocal rank fusion?+
A way to merge several ranked lists without comparing their scores. Each document scores one divided by a constant plus its rank in each list, and the sums are added. Because only positions matter, you avoid trying to normalise a cosine similarity against a keyword relevance score.
Why retrieve twenty chunks and then send four?+
The first search is fast and approximate so it can run across millions of chunks. A reranker reads the question and a candidate together and scores them accurately, but is far too slow for the whole corpus. Retrieving wide and reranking narrow combines the speed of one with the accuracy of the other.
Does RAG eliminate hallucination?+
No. It reduces it and, more importantly, makes it detectable. If every claim must cite a passage, an unsupported claim is a failure you can catch automatically in evaluation, rather than a plausible sentence nobody questions.
Should I fine-tune instead?+
They solve different problems. Fine-tuning teaches format, tone and task behaviour. Retrieval supplies facts that change, and it lets you update an answer by editing a document instead of retraining. Most production systems need retrieval first and fine-tuning only afterwards, if at all.
