LLM Basics

The robot doesn’t think. It predicts. Very confidently.

▶ the lesson · narrated sketch video · tap for sound

the intro

Large language models generate text one piece at a time, predicting what comes next from everything before it. Understanding tokens, the context window, and why temperature changes the output explains almost every weird behavior you will meet.

fun fact ✦

LLMs predict the next token from probabilities — which is why they can be both brilliant and confidently wrong.

the docs

the deep dive

the ideas behind the video — concepts, diagrams, a walkthrough and the traps to avoid

An LLM is an extremely well-read autocomplete. It has read a huge slice of the internet, and its only job is: given this text, what token probably comes next? Reasoning, code, poems — all of it is that one trick repeated thousands of times at speed. Once you see the model as prediction instead of lookup, or thinking, the weird behaviors start making sense.

prediction, not lookup

1your prompt2predict3append4repeat
one reply is this loop, run thousands of times

key terms

token

The chunk of text the model actually reads — a word, a piece of a word, or punctuation. Billing and limits count tokens, not letters.

"strawberry" ≈ 3 tokens

context window

How much the model can see at once, measured in tokens. Instructions, history, and the answer all share one budget.

200k tokens ≈ a 500-page book

temperature

The randomness dial. Low values pick the most likely token every time; high values take risks and vary the output.

0 for code, 0.7 for brainstorming

hallucination

A fluent, confident answer that is simply wrong. It is a side effect of prediction, not a bug you can patch away.

invented citations, real-looking

how it works

  1. 1
    Split into tokens

    Your text becomes tokens, the units the model reads and bills you for.

  2. 2
    Read everything at once

    The model weighs every token in the context window simultaneously. That mechanism is attention, from the 2017 transformer paper.

  3. 3
    Score the next token

    It assigns a probability to every possible next token and samples one. Temperature shapes the sampling.

  4. 4
    Append and repeat

    The chosen token joins the input and the loop runs again, until a stop condition or the token limit ends it.

the code, explained

one reply, token by token

1const ctx = tokenize(prompt)
text becomes tokens, the units the model reads
2let out: string[] = []
3while (!done(out)) {
4 const next = predict([...ctx, ...out])
scores every possible next token, then picks one
5 out.push(next)
the new token is part of the input next round
6}
7return out.join('')

side by side

low temperature

  • same prompt, near-same output
  • best for code, math, facts
  • the safe default for products
VS

high temperature

  • varied wording, fresh ideas
  • good for brainstorming
  • more chances to go off-rails
  • creative writing, ad copy
context windowsystem ruleschat historynew tokens
everything the model can see must fit in one window
🪤 gotcha

Ask a model to count the r's in "strawberry" and it can fail — not because it is dumb, but because it never saw letters, only tokens. Counting, spelling, and arithmetic live at the wrong layer of the stack. The fix is a tool that runs real code, not scolding the model.

quick answers

Is the model storing my data?

It stores nothing between calls. Each request is stateless — you resend the whole history every time, which is also why long chats get expensive.

Why does it get math wrong?

It predicts plausible-looking text, and "roughly right" reads as plausible. A calculator or code tool does the actual arithmetic.

What is a context window, in practice?

The token budget for one request: system prompt plus conversation plus answer. Overflow it and old messages get dropped or the call fails.

quick hits about LLM Basics

short answers · interviews, tips, trivia & hidden gems

🎤 interview

What is a token?watch ↗

🎤 interview

What does temperature do?watch ↗

🎤 interview

Is an LLM a search engine?watch ↗

📚 tutorial

What is a context window?watch ↗

🧠 trivia

Where did LLMs come from?watch ↗

💎 deep

Is temperature 0 deterministic?watch ↗