RAG

Give the robot your notes before the exam.

▶ the lesson · narrated sketch video · tap for sound

the intro

Retrieval-augmented generation grounds an LLM in your data: documents are split into chunks, embedded into vectors, and the chunks most similar to the question are handed to the model as context. It answers from your notes instead of its memory.

fun fact ✦

RAG is an open-book exam for AI: retrieval finds the page, the model writes the answer.

the docs

the deep dive

the ideas behind the video — concepts, diagrams, a walkthrough and the traps to avoid

A model's training data froze months ago and never included your wiki, your tickets, or last Tuesday's changelog. RAG closes that gap with an open-book exam: retrieval fetches the right pages, and the model writes the answer from them. It is cheaper and more current than retraining, and the citations tell you when the book actually got opened.

open-book exam

your docschunk+embedvector db
indexing: done once, redone whenever the docs change

key terms

embedding

A list of numbers representing meaning. Similar texts land close together in that space.

"reset password" ≈ "cannot log in"

chunking

Splitting documents into retrievable pieces. Size decides what the model gets to see.

300-500 tokens, 10-20% overlap

vector database

Stores embeddings and finds the nearest ones to any query in milliseconds. That lookup is retrieval.

Pinecone, pgvector, Qdrant

top-k

How many chunks to retrieve per question. Too few misses context, too many drowns the model in noise.

k = 3-5 is the usual start

how it works

  1. 1
    Chunk your documents

    Split into 300-500 token pieces with overlap, so answers do not get cut in half at a boundary.

  2. 2
    Embed and index

    Convert each chunk to a vector and store it next to the source text it came from.

  3. 3
    Embed the question

    The question becomes a vector in the same space as your chunks, so distance means relevance.

  4. 4
    Retrieve, answer, cite

    Grab the 3-5 nearest chunks, hand them to the model as context, and attach the chunk ids to the answer.

the code, explained

index once, retrieve per question

1for (const chunk of chunk(doc, 400)) {
about 400 tokens per chunk, plus overlap
2 const vec = await embed(chunk.text)
3 db.insert({ vec, text: chunk.text })
4}
5const qVec = await embed(question)
6const top = db.search(qVec, { k: 5 })
nearest-neighbor search, milliseconds fast
7return askLlm(question, top)
answer from the retrieved text, not memory

side by side

RAG

  • your facts, minutes fresh
  • cites the source chunks
  • no training, just indexing
  • audit-friendly answers
VS

fine-tuning

  • teaches style and format
  • needs training data + runs
  • facts go stale silently
  • no built-in citations
questionnearest chunksLLMcited answer
query time: retrieve first, answer second, cite always
🪤 gotcha

When retrieval comes back empty or irrelevant, the model does not raise an error — it quietly falls back to training memory and answers anyway. Log the retrieved chunk ids with every answer, so you can tell grounded answers from confident guesses.

quick answers

RAG or fine-tuning?

RAG for knowledge that changes and needs citations; fine-tuning for style and format. Most "the model does not know X" problems are RAG problems.

Why not paste whole documents in?

Context is a budget. Long context works, but every call re-pays for every token — retrieval pays once at index time.

How fresh is the index?

As fresh as your last embedding run. Re-index on document change — stale indexes are why "RAG said something outdated" happens.

quick hits about RAG

short answers · interviews, tips, trivia & hidden gems

🎤 interview

What are embeddings?watch ↗

📚 tutorial

What does a vector database do?watch ↗

🔧 useful

Chunk size matters more than you thinkwatch ↗

🔧 useful

How many chunks should RAG retrieve?watch ↗

💎 deep

Should RAG answers cite sources?watch ↗

💎 deep

Vector search alone is not enoughwatch ↗