These are the notes for AI episode 24. A model only knows what it saw in training, so it guesses about your shop, your company or your notes. RAG (retrieval-augmented generation) fixes that: retrieve the most relevant pieces of your documents, augment the prompt with them, and let the model generate an answer from them. You'll see why pasting everything doesn't scale, why keyword search misses synonyms, how embeddings search by meaning, how to chunk documents (size and overlap), how a threshold lets the bot say "I don't know", and real techniques: contextual retrieval, hybrid search (BM25 + embeddings) and reranking. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13). The embedding is a toy with four hand-made meaning groups; real embeddings are learned by a model (episode 13).
The problem: a confident guess
TRAINED_ON = {"refund": "Most shops accept refunds within 30 days.",
"open": "Most shops open at 10 am."} # learned from the web
def llm(question): # toy LLM: only knows what it saw in training
for topic, guess in TRAINED_ON.items():
if topic in question.lower():
return guess
return "Yes, I think so!"
print("Q: What is Ramesh's refund policy?")
print("A:", llm("What is Ramesh's refund policy?"))
print("Shop manual says: refunds only within 2 days, with the bill.")
Output:
Q: What is Ramesh's refund policy?
A: Most shops accept refunds within 30 days.
Shop manual says: refunds only within 2 days, with the bill.
The model never saw Ramesh's manual (it isn't on the internet, and it was written after training), so it answers with what is typical. A confident made-up answer is a hallucination.
Real-life example: a new employee and a 1,000-page manual
| What the employee does | In AI |
|---|---|
| Guesses: "30 days, I think?" | an LLM without your data: hallucination |
| Reads the whole manual for every question | pasting every document into the prompt |
| Uses the index, reads only the refund page | retrieval: find the relevant chunks |
| Answers from the page and shows "page 42" | a grounded answer with a citation |
Obvious fix 1: paste the whole manual
pages = 1000 # manuals, recipes, staff rules, supplier deals
tokens_per_page = 500 # rough: one page of text
questions = 500 # WhatsApp questions per day
manual = pages * tokens_per_page
print(f"whole manual : {manual:>11,} tokens per question")
print(f"pasted every time : {manual * questions:>11,} tokens per day")
relevant = 2 * 100 # 2 useful paragraphs, ~100 tokens each
print(f"only what's relevant: {relevant * questions:>11,} tokens per day")
print(f"that is {manual // relevant:,} times less")
Output:
whole manual : 500,000 tokens per question
pasted every time : 250,000,000 tokens per day
only what's relevant: 100,000 tokens per day
that is 2,500 times less
Pasting works for small documents. Anthropic's guide on contextual retrieval says that if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can just include all of it in the prompt. Ramesh's 1,000 pages are bigger, and every question would pay for all of it.
Obvious fix 2: keyword search
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
"Timings: we open at 7 am and close at 10 pm.",
"Menu: chai 15, samosa 20, vada pav 25.",
"Delivery: we do not deliver, pick up only."]
def words(text):
return set(text.lower().replace("?", " ").replace(",", " ")
.replace(".", " ").replace(":", " ").split())
def keyword_score(question, chunk): # how many words they share
return len(words(question) & words(chunk))
question = "Can I get my money back?"
for chunk in CHUNKS:
print(keyword_score(question, chunk), "|", chunk)
Output:
0 | Refunds: only within 2 days of purchase, with the bill.
0 | Timings: we open at 7 am and close at 10 pm.
0 | Menu: chai 15, samosa 20, vada pav 25.
0 | Delivery: we do not deliver, pick up only.
Why it failed: the customer said "money back", the manual says "refunds". Keyword search matches spelling, not meaning.
Search by meaning: embeddings
- An embedding is a list of numbers that captures the meaning of a text (episode 13).
- Texts with similar meaning get vectors that point in similar directions.
- Cosine similarity compares directions: 1 = same direction (same meaning), 0 = unrelated.
import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
"menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text): # toy embedding: one number per meaning group
ws = re.findall(r"[a-z]+", text.lower())
return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b): # 1.0 = same direction, 0.0 = unrelated
size = math.hypot(*a) * math.hypot(*b)
return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
"Timings: we open at 7 am and close at 10 pm.",
"Menu: chai 15, samosa 20, vada pav 25.",
"Delivery: we do not deliver, pick up only."]
question = "Can I get my money back?"
print("question:", embed(question))
for c in CHUNKS:
print(f"{cosine(embed(question), embed(c)):.2f} | {embed(c)} | {c[:26]}")
Output:
question: [2, 0, 0, 0]
1.00 | [2, 0, 0, 0] | Refunds: only within 2 day
0.00 | [0, 5, 0, 0] | Timings: we open at 7 am a
0.00 | [0, 0, 5, 0] | Menu: chai 15, samosa 20,
0.00 | [0, 0, 0, 3] | Delivery: we do not delive
This is semantic search. Real systems compute the embedding of every chunk ahead of time and store them in a vector database, built to find the closest vectors quickly, even among millions.
Retrieve, augment, generate
import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
"menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text): # toy embedding: one number per meaning group
ws = re.findall(r"[a-z]+", text.lower())
return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b): # 1.0 = same direction, 0.0 = unrelated
size = math.hypot(*a) * math.hypot(*b)
return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
"Timings: we open at 7 am and close at 10 pm.",
"Menu: chai 15, samosa 20, vada pav 25.",
"Delivery: we do not deliver, pick up only."]
def retrieve(q, k=1): # R: the most similar chunks
return sorted(CHUNKS, key=lambda c: -cosine(embed(q), embed(c)))[:k]
q = "Can I get my money back?"
print(f"Answer ONLY from the context. If it isn't there, say I don't know.\n"
f"Context: {retrieve(q)}\nQuestion: {q}")
Output:
Answer ONLY from the context. If it isn't there, say I don't know.
Context: ['Refunds: only within 2 days of purchase, with the bill.']
Question: Can I get my money back?
This prompt is sent to the model with the API call from episode 23, and the model answers "only within 2 days, with the bill" because the answer is in front of it.
question ──► embed ──► find top-k chunks (retrieve)
│
▼
instruction + chunks + question (augment) ──► LLM ──► answer (generate)
| Step | What happens |
|---|---|
| Retrieve | find the k chunks most similar to the question |
| Augment | put them in the prompt as context, with "answer only from the context" |
| Generate | the LLM writes the answer from that context |
Why it helps:
- Grounding: the answer is based on text you supplied, so the model reads instead of guessing.
- Easy updates: change the document, not the model. No retraining.
- Citations: you know which chunk was used, so you can show the source.
When nothing is relevant
import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
"menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text): # toy embedding: one number per meaning group
ws = re.findall(r"[a-z]+", text.lower())
return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b): # 1.0 = same direction, 0.0 = unrelated
size = math.hypot(*a) * math.hypot(*b)
return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
"Timings: we open at 7 am and close at 10 pm.",
"Menu: chai 15, samosa 20, vada pav 25.",
"Delivery: we do not deliver, pick up only."]
def retrieve(q, k=1): # R: the most similar chunks
return sorted(CHUNKS, key=lambda c: -cosine(embed(q), embed(c)))[:k]
q = "Do you sell pizza?"
print(f"Answer ONLY from the context. If it isn't there, say I don't know.\n"
f"Context: {retrieve(q)}\nQuestion: {q}")
Output:
Answer ONLY from the context. If it isn't there, say I don't know.
Context: ['Refunds: only within 2 days of purchase, with the bill.']
Question: Do you sell pizza?
Top-k retrieval always returns something, even when every score is 0. Irrelevant context can confuse the model. Add a minimum score (a threshold):
import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
"menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text): # toy embedding: one number per meaning group
ws = re.findall(r"[a-z]+", text.lower())
return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b): # 1.0 = same direction, 0.0 = unrelated
size = math.hypot(*a) * math.hypot(*b)
return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
CHUNKS = ["Refunds: only within 2 days of purchase, with the bill.",
"Timings: we open at 7 am and close at 10 pm.",
"Menu: chai 15, samosa 20, vada pav 25.",
"Delivery: we do not deliver, pick up only."]
def retrieve(q, k=1, min_score=0.3): # only chunks that really match
ranked = sorted(CHUNKS, key=lambda c: -cosine(embed(q), embed(c)))[:k]
return [c for c in ranked if cosine(embed(q), embed(c)) >= min_score]
for q in ["Do you sell pizza?", "When do you close?"]:
print(q, "->", retrieve(q) or "nothing found: say I don't know")
Output:
Do you sell pizza? -> nothing found: say I don't know
When do you close? -> ['Timings: we open at 7 am and close at 10 pm.']
An honest "I don't know" is better than a confident wrong answer.
Chunking: size and overlap
manual = ("Refunds are allowed only within 2 days of purchase. You must show "
"the bill. We open at 7 am and close at 10 pm. We do not deliver.")
words = manual.split()
size, overlap = 10, 0 # words per chunk, words shared
step = size - overlap
chunks = [" ".join(words[i:i + size]) for i in range(0, len(words), step)]
for n, chunk in enumerate(chunks):
print(n, "|", chunk)
Output:
0 | Refunds are allowed only within 2 days of purchase. You
1 | must show the bill. We open at 7 am and
2 | close at 10 pm. We do not deliver.
Chunk 0 ends with "You" and chunk 1 starts with "must show the bill": the rule was cut in half. Overlap repeats the last few words of the previous chunk:
manual = ("Refunds are allowed only within 2 days of purchase. You must show "
"the bill. We open at 7 am and close at 10 pm. We do not deliver.")
words = manual.split()
size, overlap = 10, 4 # words per chunk, words shared
step = size - overlap
chunks = [" ".join(words[i:i + size]) for i in range(0, len(words), step)]
for n, chunk in enumerate(chunks):
print(n, "|", chunk)
Output:
0 | Refunds are allowed only within 2 days of purchase. You
1 | days of purchase. You must show the bill. We open
2 | the bill. We open at 7 am and close at
3 | am and close at 10 pm. We do not deliver.
4 | We do not deliver.
| Chunk size | Problem |
|---|---|
| too small | a chunk loses its meaning; ideas get split |
| too big | you stuff in text that isn't needed; more tokens |
Real systems often use chunks of a few hundred tokens and split on headings and paragraphs, not just word counts.
Lost context in a chunk
import math, re
GROUPS = ["refund refunds money back bill", "open close timings am pm when",
"menu chai samosa vada pav price", "delivery deliver home pick"]
def embed(text): # toy embedding: one number per meaning group
ws = re.findall(r"[a-z]+", text.lower())
return [sum(w in g.split() for w in ws) for g in GROUPS]
def cosine(a, b): # 1.0 = same direction, 0.0 = unrelated
size = math.hypot(*a) * math.hypot(*b)
return sum(x * y for x, y in zip(a, b)) / size if size else 0.0
chunk = "Section 4.2: it is allowed only within 2 days."
q = "Can I get my money back?"
print(f"plain chunk : {cosine(embed(q), embed(chunk)):.2f}")
with_context = "Refund policy. " + chunk # add where it came from
print(f"chunk + context : {cosine(embed(q), embed(with_context)):.2f}")
Output:
plain chunk : 0.00
chunk + context : 1.00
The chunk says "it is allowed" but never says what: the heading "Refund policy" was left behind when the page was chopped. Putting a short note of context in front of the chunk before embedding it fixes the search.
Contextual retrieval (Anthropic, 19 September 2024)
Anthropic's example: a chunk saying "The company's revenue grew by 3% over the previous quarter" doesn't say which company or which quarter. Their fix is to have a model write a short context for each chunk before it is embedded. Reported results (failure to find the right chunk in the top 20):
| Method | Failed retrievals | Reduction |
|---|---|---|
| plain embeddings | 5.7% | - |
| contextual embeddings | 3.7% | 35% fewer |
| + contextual BM25 (hybrid search) | 2.9% | 49% fewer |
| + reranking | 1.9% | 67% fewer |
- BM25 is a classic keyword-ranking method; it catches exact words such as names and codes that embeddings can miss.
- Hybrid search combines keyword and meaning search.
- Reranking: a second model re-scores the top candidates and keeps the best.
- They also found that passing 20 chunks worked better than 10 or 5.
Where RAG came from
Patrick Lewis and colleagues, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", arXiv, 22 May 2020 (NeurIPS 2020). They combined:
- parametric memory: knowledge stored in a pre-trained model's weights (parameters), and
- non-parametric memory: a dense vector index of Wikipedia, searched by a neural retriever.
They reported state-of-the-art results on three open-domain question-answering tasks and more specific, diverse and factual generation, and named the problems RAG addresses: updating a model's knowledge and showing where an answer came from (provenance).
Glossary
| Term | Meaning | Example from this episode |
|---|---|---|
| RAG | retrieval-augmented generation: retrieve, augment, generate | the refund answer |
| Knowledge base | the documents the system searches | Ramesh's manual |
| Hallucination | a confident, made-up answer | "within 30 days" |
| Knowledge cutoff | the date after which a model has no training data | the manual was written later |
| Context stuffing | pasting whole documents into the prompt | 500,000 tokens a question |
| Chunk | a small piece of a document | "Refunds: only within 2 days..." |
| Chunk size | how big each chunk is | 10 words here; real: a few hundred tokens |
| Overlap | words shared by neighbouring chunks | overlap 4 |
| Keyword search | matching the same words | 0 matches for "money back" |
| Embedding | numbers that capture meaning | [2, 0, 0, 0] |
| Cosine similarity | how alike two vectors' directions are (0 to 1 here) | 1.00 for the refund chunk |
| Semantic search | search by meaning | "money back" finds "Refunds" |
| Vector database | stores embeddings and finds the closest ones fast | holds every chunk's vector |
| Retrieval / retriever | finding the most relevant chunks | retrieve(q, k=1) |
| Top-k | the k best-scoring chunks | k = 1 |
| Augment | add the retrieved chunks to the prompt | "Context: [...]" |
| Grounding | basing the answer on supplied text | "answer ONLY from the context" |
| Threshold | a minimum score to count as relevant | min_score=0.3 |
| Citation | showing which source an answer used | "page 42" |
| Contextual retrieval | adding a short context to each chunk before embedding | "Refund policy. " + chunk |
| BM25 | a classic keyword-ranking method | catches exact names and codes |
| Hybrid search | keyword + meaning search together | BM25 + embeddings |
| Reranking | a second model re-scores the top candidates | 5.7% → 1.9% failures |
| Parametric memory | what a model stores in its weights | "most shops: 30 days" |
| Non-parametric memory | an external index the model can search | a vector index of Wikipedia |
Common mistakes
| Code / situation | Exact error or symptom | Why | Fix |
|---|---|---|---|
| keyword search for "money back" | every score 0: the refund chunk is missed | same meaning, different words | embeddings or hybrid search |
retrieve("Do you sell pizza?") |
returns the refund rule | top-k always returns something | a minimum score; else "I don't know" |
cosine without the size check, on an all-zero vector |
ZeroDivisionError: float division by zero |
an empty embedding has length 0 | return 0.0 when the size is 0 |
| chunks with no overlap | a rule split across two chunks | fixed-size cuts ignore sentences | overlap; split on headings and paragraphs |
| a chunk without its heading | score 0 for the right chunk | the chunk lost its context | add the context to each chunk |
Mini project: a tiny RAG bot with citations
Give each chunk an id ("[1] Refunds: ..."). Write answer(q): retrieve the top 2 chunks above min_score=0.3; if none, return "Sorry, I don't know."; otherwise build the prompt from episode 24 and return it together with the ids of the chunks used ("Sources: [1]"). Test it on five questions, including one that isn't in the manual.
Practice
- Add a chunk
"Parking: free parking behind the shop."and ask "Where can I park?". What happens? Fix it. - Set
k=2and ask "When do you close, and is chai cheap?". Which two chunks come back? - Try overlap 2 and 6. How many chunks do you get?
- Find a question that keyword search answers but our toy embedding doesn't.
Solutions
1. With the original four meaning groups, every score is 0: the toy embedding has no "parking" meaning. Add a fifth group "park parking car bike":
0.00 | Refunds: only within 2 days of purchase, with the bill.
0.00 | Timings: we open at 7 am and close at 10 pm.
0.00 | Menu: chai 15, samosa 20, vada pav 25.
0.00 | Delivery: we do not deliver, pick up only.
1.00 | Parking: free parking behind the shop.
Real embedding models learn meanings for almost every word, so this happens far less, but rare words, product codes and names are still a weak spot (that's what BM25 helps with).
2.
Answer ONLY from the context. If it isn't there, say I don't know.
Context: ['Timings: we open at 7 am and close at 10 pm.', 'Menu: chai 15, samosa 20, vada pav 25.']
Question: When do you close, and is chai cheap?
3. Overlap 2 gives 4 chunks; overlap 6 gives 7 chunks:
0 | Refunds are allowed only within 2 days of purchase. You
1 | purchase. You must show the bill. We open at 7
2 | at 7 am and close at 10 pm. We do
3 | We do not deliver.
0 | Refunds are allowed only within 2 days of purchase. You
1 | within 2 days of purchase. You must show the bill.
2 | purchase. You must show the bill. We open at 7
3 | the bill. We open at 7 am and close at
4 | at 7 am and close at 10 pm. We do
5 | close at 10 pm. We do not deliver.
6 | We do not deliver.
More overlap means more chunks (more storage and embedding work) but fewer ideas cut in half.
4. With the parking chunk added, ask "Is it free?". Keyword search scores the parking chunk 1 (the shared word "free"); the toy embedding scores every chunk 0.00, because "free" is in none of its meaning groups. This is why real systems combine both (hybrid search).
Quick quiz 1
Ramesh changes refunds from 2 days to 3. How does the RAG bot learn it? A: retrain the model · B: edit the document in the knowledge base · C: raise max_tokens · D: it can't
Answer
B. RAG reads the documents at question time; update the document and the next answer uses it.
Quick quiz 2
Question: "What time do you shut?" Chunk: "Timings: we close at 10 pm." Which search finds it? A: keyword search · B: meaning search with embeddings · C: neither · D: only the LLM, no search
Answer
B. "Shut" and "close" are different words with the same meaning.
Interview questions
- What is RAG and why use it instead of fine-tuning? Retrieve relevant documents at query time and put them in the prompt; knowledge is easy to update, answers are grounded and can cite sources, and no retraining is needed.
- Walk through a RAG pipeline. Split documents into chunks, embed and index them in a vector store; at query time embed the question, retrieve top-k (optionally hybrid + rerank), build a prompt with the context and instructions, generate, and return citations.
- How do you choose chunk size and overlap? Trade-off between context and precision; typically a few hundred tokens, split on structure (headings, paragraphs), with some overlap; evaluate retrieval quality to tune.
- What happens when nothing relevant is retrieved? Top-k still returns chunks; use a similarity threshold and instruct the model to say it doesn't know.
- What is hybrid search and why use it? Combining keyword ranking (e.g. BM25) with embeddings: embeddings catch synonyms, BM25 catches exact names, numbers and codes.
- What is reranking? A second, more accurate model re-scores the retrieved candidates so the best ones go into the prompt.
- When is RAG unnecessary? When the whole knowledge base fits comfortably in the prompt (Anthropic's guide: under about 200,000 tokens).
Summary
- A model doesn't know your documents, so it guesses (hallucination).
- Pasting everything is fine for small knowledge bases, but it doesn't scale.
- Keyword search misses synonyms; embeddings + cosine similarity search by meaning.
- RAG = retrieve the best chunks, augment the prompt, generate the answer.
- A threshold lets the bot say "I don't know"; citations show the source.
- Good chunks need sensible size, overlap and context; hybrid search and reranking find them more reliably.
- Next episode: evaluating AI apps.