RAG
Give the robot your notes before the exam.
▶ the lesson · narrated sketch video · tap for sound
the intro
Retrieval-augmented generation grounds an LLM in your data: documents are split into chunks, embedded into vectors, and the chunks most similar to the question are handed to the model as context. It answers from your notes instead of its memory.
fun fact ✦
RAG is an open-book exam for AI: retrieval finds the page, the model writes the answer.
the docs
the deep dive
the ideas behind the video — concepts, diagrams, a walkthrough and the traps to avoid
A model's training data froze months ago and never included your wiki, your tickets, or last Tuesday's changelog. RAG closes that gap with an open-book exam: retrieval fetches the right pages, and the model writes the answer from them. It is cheaper and more current than retraining, and the citations tell you when the book actually got opened.
open-book exam
key terms
embedding
A list of numbers representing meaning. Similar texts land close together in that space.
"reset password" ≈ "cannot log in"chunking
Splitting documents into retrievable pieces. Size decides what the model gets to see.
300-500 tokens, 10-20% overlapvector database
Stores embeddings and finds the nearest ones to any query in milliseconds. That lookup is retrieval.
Pinecone, pgvector, Qdranttop-k
How many chunks to retrieve per question. Too few misses context, too many drowns the model in noise.
k = 3-5 is the usual starthow it works
- 1Chunk your documents
Split into 300-500 token pieces with overlap, so answers do not get cut in half at a boundary.
- 2Embed and index
Convert each chunk to a vector and store it next to the source text it came from.
- 3Embed the question
The question becomes a vector in the same space as your chunks, so distance means relevance.
- 4Retrieve, answer, cite
Grab the 3-5 nearest chunks, hand them to the model as context, and attach the chunk ids to the answer.
the code, explained
index once, retrieve per question
side by side
RAG
- your facts, minutes fresh
- cites the source chunks
- no training, just indexing
- audit-friendly answers
fine-tuning
- teaches style and format
- needs training data + runs
- facts go stale silently
- no built-in citations
When retrieval comes back empty or irrelevant, the model does not raise an error — it quietly falls back to training memory and answers anyway. Log the retrieved chunk ids with every answer, so you can tell grounded answers from confident guesses.
quick answers
RAG or fine-tuning?
RAG for knowledge that changes and needs citations; fine-tuning for style and format. Most "the model does not know X" problems are RAG problems.
Why not paste whole documents in?
Context is a budget. Long context works, but every call re-pays for every token — retrieval pays once at index time.
How fresh is the index?
As fresh as your last embedding run. Re-index on document change — stale indexes are why "RAG said something outdated" happens.
quick hits about RAG
short answers · interviews, tips, trivia & hidden gems