What is RAG?
RAG stands for retrieval-augmented generation. Before the language model answers, your code searches your own documents. It then pastes the most relevant passages into the prompt. The model answers from those passages instead of from memory.
Think of it as an open-book exam. A closed-book student guesses when they forget. An open-book student first finds the right page, then writes the answer and can point to where it came from.
The name comes from a 2020 research paper by Patrick Lewis and colleagues, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv, 22 May 2020). Today the term covers any system that retrieves text and then generates an answer from it.
RAG solves two problems a plain chatbot has:
- It does not know your data. No model was trained on your company handbook, your course notes or yesterday's tickets.
- It makes things up when it does not know. Grounding the answer in retrieved text, and asking for a citation, makes a wrong answer much easier to catch.
How RAG works, in four steps
- Chunk. Split your documents into small passages, like one section or a few paragraphs each.
- Index. Make the chunks searchable. Either count words (keyword search) or turn each chunk into a list of numbers that captures its meaning (an embedding).
- Retrieve. When a question comes in, find the few chunks that best match it.
- Generate. Put those chunks into the prompt with the question and an instruction: "answer only from this context, cite the chunk".
Steps 1 and 2 run once, when documents change. Steps 3 and 4 run on every question. Most RAG quality problems are retrieval problems: if step 3 finds the wrong chunk, the best model in the world still answers wrong.
Build a small RAG pipeline in Python, for free
We will build the whole pipeline on a laptop with no paid API. The data is a short, made-up intern handbook for a fictional company, with eight sections: leave, sick days, stipend, laptop, working hours, certificate, travel and pre-placement offer. A made-up handbook is useful here because no model can know its answers in advance.
What we used: Python 3.12 with only the standard library, and Ollama 0.24.0 to run two free models locally: nomic-embed-text (embeddings) and qwen2.5:3b (answers). Install Ollama, then run ollama pull nomic-embed-text and ollama pull qwen2.5:3b. Save each file below in one folder, next to handbook.md.
handbook.md (first two sections)
# Leave
Interns get 1.5 days of paid leave for every completed month. Unused leave lapses when the internship ends; it is not paid out. Apply for leave in the HR portal at least 3 working days before the first day off.
# Sick days
If you are unwell, message your mentor before 10:00 on the same day. Up to 2 sick days a month do not need a doctor's note. A third sick day in the same month needs a medical certificate uploaded to the HR portal.
# Stipend
The monthly stipend is Rs 25,000. It is paid on the 7th of the next month into the bank account on form HR-17. If the 7th is a bank holiday, it is paid on the previous working day.
Step 1: split the handbook into chunks
Our handbook already has one heading per topic, so each section becomes one chunk. Real documents need more care: split long sections into pieces of a few hundred words, with a little overlap so a sentence is never cut away from its context.
chunk.py
import re
def load_chunks(path="handbook.md"):
text = open(path, encoding="utf-8").read()
chunks = []
for block in text.split("# ")[1:]:
title, _, body = block.partition("\n")
chunks.append({"id": len(chunks), "title": title.strip(), "text": body.strip()})
return chunks
if __name__ == "__main__":
chunks = load_chunks()
print(len(chunks), "chunks")
for c in chunks:
print(c["id"], c["title"], "-", len(c["text"].split()), "words")
Output
8 chunks
0 Leave - 40 words
1 Sick days - 43 words
2 Stipend - 38 words
3 Laptop - 36 words
4 Working hours - 30 words
5 Certificate - 32 words
6 Travel - 35 words
7 Pre-placement offer - 36 words
Step 2: keyword search with BM25
BM25 is a classic keyword-ranking formula, and the default similarity in Elasticsearch. It scores a chunk higher when it contains the question's words, gives rare words more weight than common ones, and stops rewarding a word after it repeats a few times. It comes from the probabilistic relevance framework described by Robertson and Zaragoza (2009).
bm25.py
import math, re
from collections import Counter
from chunk import load_chunks
STOP = {"a", "an", "the", "is", "are", "i", "my", "do", "if", "of", "to", "in",
"for", "on", "and", "or", "it", "you", "your", "get", "what", "when",
"how", "can", "will", "be", "after", "which", "with", "from", "at"}
def tokens(text):
return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]
class BM25:
def __init__(self, docs, k1=1.5, b=0.75):
self.docs = [tokens(d) for d in docs]
self.k1, self.b = k1, b
self.avgdl = sum(len(d) for d in self.docs) / len(self.docs)
df = Counter(w for d in self.docs for w in set(d))
n = len(self.docs)
self.idf = {w: math.log((n - f + 0.5) / (f + 0.5) + 1) for w, f in df.items()}
def scores(self, query):
out = []
for d in self.docs:
tf, s = Counter(d), 0.0
for w in tokens(query):
if w in tf:
f = tf[w]
s += self.idf[w] * f * (self.k1 + 1) / (
f + self.k1 * (1 - self.b + self.b * len(d) / self.avgdl))
out.append(s)
return out
if __name__ == "__main__":
chunks = load_chunks()
bm25 = BM25([c["title"] + " " + c["text"] for c in chunks])
for q in ["When is the stipend paid?", "Can I work from home?"]:
ranked = sorted(zip(bm25.scores(q), chunks), key=lambda x: -x[0])[:3]
print(q)
for s, c in ranked:
print(f" {s:5.2f} {c['title']}")
Output
When is the stipend paid?
4.57 Stipend
1.75 Leave
0.00 Sick days
Can I work from home?
1.86 Travel
1.31 Working hours
0.00 Leave
The first question works: "stipend" appears in exactly one chunk. The second fails. The handbook says "Fridays are remote", not "work from home", so BM25 matches the word "work" in the Travel section instead. Keyword search only sees words, not meaning.
Step 3: meaning search with embeddings
An embedding turns a piece of text into a long list of numbers, so that texts with similar meaning get similar lists. You can then compare a question with every chunk using cosine similarity, which is 1.0 for the same direction and lower as meanings drift apart.
The nomic-embed-text model card asks for a prefix on every input: search_document: for chunks and search_query: for questions. Skipping the prefixes quietly lowers retrieval quality.
embed.py
import json, math, urllib.request
from chunk import load_chunks
OLLAMA = "http://localhost:11434/api/embed"
def embed(texts, model="nomic-embed-text"):
body = json.dumps({"model": model, "input": texts}).encode()
req = urllib.request.Request(OLLAMA, body, {"Content-Type": "application/json"})
return json.load(urllib.request.urlopen(req))["embeddings"]
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
class VectorIndex:
def __init__(self, docs):
# nomic-embed-text expects task prefixes on documents and queries
self.vecs = embed(["search_document: " + d for d in docs])
def scores(self, query):
q = embed(["search_query: " + query])[0]
return [cosine(q, v) for v in self.vecs]
if __name__ == "__main__":
chunks = load_chunks()
index = VectorIndex([c["title"] + " " + c["text"] for c in chunks])
print("dimensions:", len(index.vecs[0]))
for q in ["When is the stipend paid?", "Can I work from home?"]:
ranked = sorted(zip(index.scores(q), chunks), key=lambda x: -x[0])[:3]
print(q)
for s, c in ranked:
print(f" {s:.3f} {c['title']}")
Output
dimensions: 768
When is the stipend paid?
0.819 Stipend
0.631 Leave
0.580 Travel
Can I work from home?
0.570 Travel
0.553 Working hours
0.538 Leave
Embeddings put Stipend far ahead for the first question (0.819 against 0.631). But they also fail "Can I work from home?": Travel scores 0.570 and Working hours 0.553. The gap is tiny, and the vocabulary mismatch still wins. We ran this twice and got the same numbers both times.
Step 4: measure retrieval before you trust it
Never judge retrieval on two hand-picked questions. Write a small test set of questions with the chunk that should answer each, and count how often the top result is right. We also test a hybrid: combine both rankings with reciprocal rank fusion (RRF), where each ranking gives a chunk 1 / (60 + rank) points. This method comes from Cormack, Clarke and Büttcher (SIGIR 2009).
evaluate.py
from chunk import load_chunks
from bm25 import BM25
from embed import VectorIndex
TESTS = [
("When does the stipend get credited?", "Stipend"),
("How much money do I get per month?", "Stipend"),
("Which form has my bank details?", "Stipend"),
("What if my laptop gets stolen?", "Laptop"),
("Can I work from home?", "Working hours"),
("Do I need a doctor's note if I am ill?", "Sick days"),
("Will I get a full-time job after the internship?", "Pre-placement offer"),
("How do I get reimbursed for an Uber ride?", "Travel"),
("What happens to leave I do not use?", "Leave"),
("When will I receive my internship certificate?", "Certificate"),
]
def ranking(scores):
return sorted(range(len(scores)), key=lambda i: -scores[i])
def rrf(*rankings, k=60):
# reciprocal rank fusion: each list votes 1 / (k + rank)
fused = {}
for r in rankings:
for rank, i in enumerate(r, start=1):
fused[i] = fused.get(i, 0) + 1 / (k + rank)
return sorted(fused, key=lambda i: -fused[i])
if __name__ == "__main__":
chunks = load_chunks()
docs = [c["title"] + " " + c["text"] for c in chunks]
bm25, vec = BM25(docs), VectorIndex(docs)
hits = {"BM25": 0, "Vectors": 0, "Hybrid": 0}
for q, want in TESTS:
b, v = ranking(bm25.scores(q)), ranking(vec.scores(q))
tops = {"BM25": b[0], "Vectors": v[0], "Hybrid": rrf(b, v)[0]}
row = []
for name, top in tops.items():
ok = chunks[top]["title"] == want
hits[name] += ok
row.append(f"{name}={'ok' if ok else chunks[top]['title']}")
print(f"{q[:44]:<44} " + ", ".join(row))
print({name: f"{n}/{len(TESTS)}" for name, n in hits.items()})
Output
When does the stipend get credited? BM25=ok, Vectors=ok, Hybrid=ok
How much money do I get per month? BM25=Sick days, Vectors=ok, Hybrid=ok
Which form has my bank details? BM25=ok, Vectors=ok, Hybrid=ok
What if my laptop gets stolen? BM25=ok, Vectors=ok, Hybrid=ok
Can I work from home? BM25=Travel, Vectors=Travel, Hybrid=Travel
Do I need a doctor's note if I am ill? BM25=ok, Vectors=ok, Hybrid=ok
Will I get a full-time job after the interns BM25=Certificate, Vectors=Leave, Hybrid=Certificate
How do I get reimbursed for an Uber ride? BM25=ok, Vectors=ok, Hybrid=ok
What happens to leave I do not use? BM25=ok, Vectors=ok, Hybrid=ok
When will I receive my internship certificat BM25=ok, Vectors=ok, Hybrid=ok
{'BM25': '7/10', 'Vectors': '8/10', 'Hybrid': '8/10'}
On these 10 questions, BM25 got the right chunk first 7 times, embeddings 8 times, and the hybrid 8 times. Two lessons:
- Embeddings rescued "How much money do I get per month?", which shares no useful word with "stipend". That is the case they are built for.
- The hybrid did not beat embeddings here. Hybrid search is a common default because keyword search catches exact codes and names, like "HR-17". It still has to earn its place on your test set, and 10 questions is far too few to choose a winner.
Two questions failed for every method. "Work from home" hits the word "remote", and "full-time job" hits "pre-placement offer". The fix is not a cleverer formula. Write the documents in your users' words (add "work from home" to the Working hours section), or have a model rewrite the question before searching.
Step 5: generate an answer from the retrieved chunks
Now we send the top two chunks to qwen2.5:3b with strict instructions: answer only from the context, cite the chunk number, and admit when the answer is not there. We use temperature 0 and a fixed seed so the run repeats. We also ask the same question without RAG to see what the model says on its own.
answer.py
import json, urllib.request
from chunk import load_chunks
from bm25 import BM25
from embed import VectorIndex
from evaluate import ranking, rrf
def chat(prompt, model="qwen2.5:3b"):
body = json.dumps({"model": model, "stream": False,
"options": {"temperature": 0, "seed": 42},
"messages": [{"role": "user", "content": prompt}]}).encode()
req = urllib.request.Request("http://localhost:11434/api/chat", body,
{"Content-Type": "application/json"})
return json.load(urllib.request.urlopen(req))["message"]["content"]
def build_prompt(question, retrieved):
context = "\n\n".join(f"[{c['id']}] {c['title']}: {c['text']}" for c in retrieved)
return ("Answer the question using ONLY the context below. Cite the chunk "
"number like [2]. If the context does not contain the answer, say "
"\"The handbook does not say.\"\n\n"
f"Context:\n{context}\n\nQuestion: {question}")
chunks = load_chunks()
docs = [c["title"] + " " + c["text"] for c in chunks]
bm25, vec = BM25(docs), VectorIndex(docs)
def rag_answer(question, k=2):
top = rrf(ranking(bm25.scores(question)), ranking(vec.scores(question)))[:k]
retrieved = [chunks[i] for i in top]
print("retrieved:", [c["title"] for c in retrieved])
return chat(build_prompt(question, retrieved))
q = "How much is the intern stipend and when is it paid?"
print("WITHOUT RAG:", chat(q))
print("WITH RAG:", rag_answer(q))
print("NOT IN DOCS:", rag_answer("Is lunch free in the office canteen?"))
Output (qwen2.5:3b via Ollama 0.24.0, temperature 0, seed 42; your wording may differ)
WITHOUT RAG: Internship stipends can vary significantly depending on the company, industry, location, and other factors. Some companies might offer a stipend or allowance to cover basic living expenses during the internship period, while others may not provide any financial support.
The timing of payment also varies widely. It could be paid monthly, bi-weekly, weekly, or even at the end of each internship period. In some cases, interns might receive their stipends in the form of a gift card to local stores or restaurants rather than cash.
For specific information about an intern's stipend and payment schedule, I would recommend contacting the company directly where you are interested in interning. They will be able to provide you with detailed information tailored to their internship program.
retrieved: ['Stipend', 'Leave']
WITH RAG: The intern stipend is Rs 25,000 per month. It is paid on the 7th of the next month into the bank account on form HR-17. If the 7th falls on a bank holiday, it will be paid on the previous working day.
retrieved: ['Leave', 'Stipend']
NOT IN DOCS: The handbook does not say.
Three things to notice in this real run:
- Without RAG, the model cannot know the handbook, so it gives a long, generic non-answer. A bigger model might instead invent a confident number, which is worse.
- With RAG, it gives the right amount (Rs 25,000), the right date (the 7th of the next month) and the bank-holiday rule, all from the Stipend chunk. But it did not add the [2] citation we asked for. Small models often skip formatting instructions, so check citations in code instead of trusting them.
- For the canteen question, retrieval returned two irrelevant chunks (Leave and Stipend). The model still correctly said "The handbook does not say." Retrieval always returns something; the instruction to admit a gap is what stops a made-up answer.
On a laptop shared with other heavy jobs, this script took several minutes. On an idle machine a 3B model answers much faster, and a hosted API faster still.
From RAG to agentic RAG
The pipeline above always retrieves, exactly once, before answering. Agentic RAG turns retrieval into a tool that the model can choose to call. The model decides whether to search, what to search for, and whether to search again after reading the first results.
That helps with questions like "Can I take leave in the week my stipend is paid?". The agent can search for leave rules, then for the stipend date, then combine both. A single fixed retrieval might find only one of the two.
The cost is more model calls per question, more latency and more ways to go wrong. A good rule is to start with plain RAG plus a test set, and add agent steps only for the question types plain RAG keeps failing.
RAG vs fine-tuning vs a long prompt
| RAG | Fine-tuning | Paste everything into the prompt | |
|---|---|---|---|
| Best for | Facts that change, large document sets, answers that need a citation | Style, format and narrow skills the model does badly | A few documents that fit in the model's context window |
| Updating knowledge | Re-index the changed document | Train again | Edit the prompt |
| Shows its source | Yes, the retrieved chunks | No | Partly; the model may still blend sources |
| Main failure | Wrong chunk retrieved | Learns the style but still invents facts | Cost and speed grow with every token, every call |
These are not either-or. Many production systems fine-tune a small model for format and use RAG for facts.
Honest limits
- Garbage in, garbage out. If the handbook is out of date, RAG confidently quotes the old rule. Retrieval does not check whether a document is true.
- Retrieval misses are silent. In our test, two of ten questions retrieved the wrong chunk with every method, and nothing in the output warned us. Only a test set shows this.
- Small models still slip. A 3-billion-parameter model can ignore an instruction, mix two chunks, or cite the wrong number. Check the cited chunk, not just the answer.
- Prompt injection. Retrieved text goes straight into the prompt. A document that says "ignore your instructions" is an attack. Treat retrieved text as data, and never give a RAG bot tools that act on its own.
- Access control. If the index holds HR files, anyone who can ask questions can read them. Filter chunks by the user's permissions before they reach the prompt.
- Our numbers are tiny. Eight chunks and ten questions teach the method; they prove nothing about which retriever is better for your data.
Frequently asked questions
What is RAG in AI, in simple words?
RAG means the AI looks things up before it answers. Your code searches your documents for the passages that match the question, adds them to the prompt, and the model answers from them. It is like an open-book exam, and the answer can point to the page it came from.
What is the full form of RAG?
RAG stands for retrieval-augmented generation. 'Retrieval' is the search step that finds relevant text, and 'generation' is the language model writing the answer. The term comes from a 2020 paper by Patrick Lewis and colleagues, 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks'.
Do I need a vector database for RAG?
Not to start. For a few thousand chunks, a Python list of embeddings and a cosine-similarity loop, as in this guide, is enough. A vector database helps when you have millions of chunks, need filtering by permissions, or must update the index while it serves traffic.
Is RAG better than fine-tuning?
They solve different problems. RAG is better for facts that change and answers that must cite a source, because you only re-index a document. Fine-tuning is better for teaching a style, format or narrow skill. Many systems use both: a tuned model for format, RAG for facts.
Can I build RAG for free?
Yes. This guide runs fully on a laptop: Python's standard library, Ollama, the nomic-embed-text embedding model and the qwen2.5:3b chat model. There is no API bill. Expect slower answers than a hosted model, and a small model that slips more often.
What is agentic RAG?
In agentic RAG, retrieval is a tool the model can call when it decides it needs to, as many times as it needs. It can search, read, search again with a better query, and then answer. It handles multi-part questions better, at the cost of more model calls and latency.
Crio AI Engineering
Build agents that work outside a demo
Crio's AI Engineering program is project-first: you build LLM apps, RAG pipelines and tool-using agents, then learn to evaluate, secure and ship them. Talk to the team about whether it fits you.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401, 22 May 2020) · arXiv
- Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009) · Foundations and Trends in Information Retrieval
- Cormack, Clarke and Büttcher, Reciprocal rank fusion outperforms Condorcet and individual rank learning methods (SIGIR 2009) · ACM
- Elasticsearch similarity settings (BM25 is the default) · Elastic
- nomic-embed-text-v1.5 model card (task prefixes) · Nomic AI
- Ollama API: /api/embed and /api/chat · Ollama
- qwen2.5 on the Ollama library · Ollama
All code on this page was run on an Apple M1 Pro laptop on 19 November 2026 with Python 3.12.8 and Ollama 0.24.0, and every Output block is the real output. The intern handbook is invented for teaching. The model's wording will differ on other machines and versions; the retrieval scores and test counts are deterministic.
Last updated: