These are the notes for AI episode 15. You have built every part: tokens (episode 12), embeddings (13) and the Transformer (14). Here you put them to work. You build a tiny language model by counting, make it write by predicting, appending and repeating, watch greedy decoding loop forever and a longer context crash on text it never saw, then replace counting with a network that outputs a probability for every token, train it for free with self-supervised learning and the −log(p) loss, and scale it up into a large language model like GPT-3. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).
Real-life example: your phone keyboard
| On your phone | In a language model |
|---|---|
| "Happy birthday ..." → it suggests "to", then "you" | Next-token prediction: guess what comes next |
| It learned from what people type | Training text: where the guesses come from |
| Keep tapping the middle suggestion: it writes whole sentences | Generation: predict, append, repeat |
The one job
A language model takes the tokens so far, the context, and outputs a probability for every token in the vocabulary: how likely is each one to come next?
"raju drinks" → model → chai 0.67, coffee 0.33
A counting model (a bigram model)
from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju ."""
words = text.split()
follows = defaultdict(Counter) # which word comes after each word?
for a, b in zip(words, words[1:]):
follows[a][b] += 1
for w in ["drinks", "likes"]:
total = sum(follows[w].values())
print(f"after '{w}':", {n: round(c / total, 2) for n, c in follows[w].items()})
Output:
after 'drinks': {'chai': 0.67, 'coffee': 0.33}
after 'likes': {'chai': 0.5, 'cricket': 0.5}
Counting which word follows which, and dividing by the total, is already a real language model. One that looks at only the previous word is a bigram model; one that looks at the previous n − 1 words is an n-gram model.
Predict, append, repeat
from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju ."""
words = text.split()
follows = defaultdict(Counter)
for a, b in zip(words, words[1:]):
follows[a][b] += 1
word, story = "raju", ["raju"]
for _ in range(15): # predict, append, repeat
word = follows[word].most_common(1)[0][0]
story.append(word)
print(" ".join(story))
Output:
raju drinks chai . neha drinks chai . neha drinks chai . neha drinks chai .
Always taking the single most likely token is greedy decoding. With one word of memory, the model has no idea what it already said: after "." it always says "neha", after "neha" always "drinks". It loops forever.
More context, more problems
from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju ."""
words = text.split()
follows = defaultdict(Counter) # now: the last TWO words -> next word
for a, b, c in zip(words, words[1:], words[2:]):
follows[(a, b)][c] += 1
print("seen 2-word contexts:", len(follows))
print(f"possible with GPT-2's vocabulary: {50257 ** 2:,}")
for context in [("neha", "drinks"), ("raju", "plays")]:
print(context, "->", follows[context].most_common(1)[0][0])
Output:
seen 2-word contexts: 20
possible with GPT-2's vocabulary: 2,525,766,049
('neha', 'drinks') -> coffee
IndexError: list index out of range
"neha drinks" → coffee works, but "raju plays" never appeared, so there is nothing to count and the program crashes. With GPT-2's 50,257-token vocabulary there are 2,525,766,049 possible two-word contexts and about 127 trillion three-word ones (50,257³ = 126,937,424,324,593). No text contains them all: counting cannot generalize to things it never saw. This is the sparsity problem.
Replace counting with a network that generalizes
tokens (ep 12) → embeddings (ep 13) → Transformer blocks (ep 14) → one score per token → softmax → probabilities
- Similar words have similar embeddings, so what the model learns about "neha plays" carries over to "raju plays".
- Attention looks back over the whole context, not just one or two words.
- The last layer gives one score per vocabulary token (50,257 for GPT-2). Raw scores are called logits; softmax turns them into a probability distribution.
import math
vocab = ["chai", "coffee", "cricket", "neha", "."]
scores = [3.1, 2.0, 0.4, -0.5, 1.2] # pretend output of a Transformer for "neha drinks"
e = [math.exp(s) for s in scores]
probs = [x / sum(e) for x in e] # softmax, episode 14
for w, p in sorted(zip(vocab, probs), key=lambda t: -t[1]):
print(f"{w:8} {p:.2f} " + "#" * round(p * 40))
print("total:", round(sum(probs), 2))
Output:
chai 0.63 #########################
coffee 0.21 ########
. 0.09 ####
cricket 0.04 ##
neha 0.02 #
total: 1.0
Training for free: self-supervised learning
| A school workbook | Training an LLM |
|---|---|
| Any page of any book: "Raju drinks hot ____" | Training text: web pages, books, code |
| The answer key is just the next word in the book | Free labels: the next token |
| No teacher needed to mark millions of pages | Self-supervised learning: labels come from the data |
text = "raju drinks hot chai every morning before college"
tokens = text.split()
pairs = [(tokens[:i], tokens[i]) for i in range(1, len(tokens))]
for context, target in pairs:
print(f"{' '.join(context):45} -> {target}")
print(len(tokens), "tokens gave", len(pairs), "free training examples")
Output:
raju -> drinks
raju drinks -> hot
raju drinks hot -> chai
raju drinks hot chai -> every
raju drinks hot chai every -> morning
raju drinks hot chai every morning -> before
raju drinks hot chai every morning before -> college
8 tokens gave 7 free training examples
A text of N tokens gives N − 1 next-token examples. Thanks to the causal mask (episode 14), a Transformer learns all of them in one pass: each position may only look at the tokens before it.
The loss: −log(p)
import math
# the loss for one prediction: -log(probability given to the right token)
for p in [0.9, 0.5, 0.1, 0.01]:
print(f"model gave the right token {p:4}: loss = {-math.log(p):.2f}")
Output:
model gave the right token 0.9: loss = 0.11
model gave the right token 0.5: loss = 0.69
model gave the right token 0.1: loss = 2.30
model gave the right token 0.01: loss = 4.61
The loss for one prediction is minus the log of the probability the model gave to the right token. Confident and right → near 0; confident and wrong → large. Averaged over many tokens this is the cross-entropy loss. Gradient descent (episode 7) then nudges every weight to lower it, so right tokens get more probability next time.
Pretraining at scale: GPT-3
| Fact (Brown et al., OpenAI, 2020, arXiv 2005.14165) | Value |
|---|---|
| Type | "an autoregressive language model" |
| Parameters | 175 billion (≈ 350 GB at 2 bytes each) |
| Training data | 300 billion tokens |
| Transformer blocks | 96 layers, d_model 12,288, 96 attention heads |
| Context window | 2,048 tokens |
Training a model on a huge pile of general text with next-token prediction is pretraining. For comparison, GPT-2 small has 12 layers, 768 numbers per token, 12 heads and a 1,024-token context.
Why "large"? And how it writes
- Large = billions of parameters, trained on hundreds of billions of tokens, using huge compute (thousands of GPUs, episode 11).
- When you chat, it runs our loop: prompt → probabilities → pick a token → append → repeat until done. Generating one token at a time, each fed back as input, is autoregressive generation.
What an LLM is not
| Myth | Reality |
|---|---|
| A database of the web | patterns squeezed into weights; no stored copy of documents |
| It looks things up | not unless you give it tools (episode 28) or documents (RAG, episode 24) |
| An assistant by default | a pretrained base model only continues text; ask a question and it may write more questions (episode 17) |
| Always right | it predicts likely text, not checked facts (episode 19) |
Glossary
| Term | Meaning | Example |
|---|---|---|
| Language model | predicts the next token from the tokens so far | phone keyboard suggestions |
| Context | the tokens the model is given | "raju drinks" |
| Next-token prediction | the model's one job | "drinks" → chai |
| Probability | a chance from 0 to 1 | chai 0.67 |
| Probability distribution | a probability for every option, adding to 1 | chai 0.63, coffee 0.21, … |
| Bigram model | predicts from the previous word only | our counting model |
| n-gram model | predicts from the previous n − 1 words | the two-word version |
| Generation | producing text token by token | predict, append, repeat |
| Greedy decoding | always pick the most likely token | most_common(1) |
| Sparsity | most possible contexts never appear in any text | 2,525,766,049 two-word contexts |
| Generalize | make good guesses on unseen inputs | "raju plays" like "neha plays" |
| Logits | raw scores before softmax | [3.1, 2.0, 0.4, -0.5, 1.2] |
| Softmax | scores → probabilities (episode 14) | math.exp(s) / sum |
| Self-supervised learning | labels come from the data itself | the next word is the answer |
| Training example | a (context, next token) pair | "raju drinks hot" → chai |
| Loss | how wrong a prediction is | −log(0.9) = 0.11 |
| Cross-entropy | the average −log(p) over many tokens | the LLM training loss |
| Pretraining | training on huge general text by next-token prediction | GPT-3: 300 billion tokens |
| Parameters | the learned numbers (weights) | GPT-3: 175 billion |
| LLM | large language model: billions of parameters | GPT-3 |
| Autoregressive generation | each new token is fed back as input | ChatGPT writing a reply |
| Base model | a model straight out of pretraining | continues text, not an assistant |
| Context window | how many tokens it can look back over | GPT-3: 2,048 |
| End-of-text token | a special token marking the end | GPT-2's <|endoftext|> |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| A context the counting model never saw | follows[("raju", "plays")].most_common(1)[0] |
IndexError: list index out of range |
| A probability of exactly 0 in the loss | -math.log(0.0) |
ValueError: math domain error |
| Asking what follows the last word | max(follows["college"], key=...) |
ValueError: max() iterable argument is empty |
| Greedy decoding for long text | most_common(1) in a loop |
no error; it loops forever |
| Thinking more n-gram context fixes it | 3-word contexts | no error; even sparser, more unseen contexts |
Fixes: use a model that generalizes (or back off to a shorter context); never hard-code a 0 chance (softmax never gives exactly 0); add an end-of-text token; sample instead of greedy (episode 16).
Mini project: a "Raju story" generator
Collect 30 sentences about your own day, build the two-word counting model, and generate from every two-word context you can. Then fall back to the one-word model whenever a two-word context is unseen (this trick is called back-off), so it never crashes.
Practice
- Add five sentences to the counting model and generate again. Does the loop change?
- Generate from "neha" instead of "raju".
- How many training examples does a 20-word sentence give? Check it in code.
- Compute the loss when the model gives the right token p = 0.25.
Solutions
1.
from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju .
raju drinks lassi after cricket . neha reads a book . raju reads the news .
neha drinks water after lunch . raju plays chess with neha ."""
words = text.split()
follows = defaultdict(Counter)
for a, b in zip(words, words[1:]):
follows[a][b] += 1
word, story = "raju", ["raju"]
for _ in range(15): # predict, append, repeat
word = follows[word].most_common(1)[0][0]
story.append(word)
print(" ".join(story))
raju drinks chai . neha drinks chai . neha drinks chai . neha drinks chai .
Still a loop: "drinks" → chai and "." → neha are still the most common next words. More text doesn't fix greedy decoding with one word of memory.
2.
from collections import Counter, defaultdict
text = """raju drinks chai . neha drinks coffee . raju drinks chai with neha .
neha likes chai . raju likes cricket . neha plays cricket with raju ."""
words = text.split()
follows = defaultdict(Counter)
for a, b in zip(words, words[1:]):
follows[a][b] += 1
word, story = "neha", ["neha"]
for _ in range(15): # predict, append, repeat
word = follows[word].most_common(1)[0][0]
story.append(word)
print(" ".join(story))
neha drinks chai . neha drinks chai . neha drinks chai . neha drinks chai .
3.
text = "on monday raju woke up made masala chai for neha and then took the bus to his college in pune"
tokens = text.split()
pairs = [(tokens[:i], tokens[i]) for i in range(1, len(tokens))]
print(len(tokens), "tokens gave", len(pairs), "free training examples")
20 tokens gave 19 free training examples
4.
import math
print(f"loss when p = 0.25: {-math.log(0.25):.2f}")
loss when p = 0.25: 1.39
Quick quiz 1
One step of an LLM, given "neha drinks". What comes out? A: the whole answer at once · B: one probability per vocabulary token · C: a web search result · D: the single word "chai"
Answer
B. One step gives a probability distribution over the whole vocabulary; one token is then picked and the loop repeats.
Quick quiz 2
A training text of 1,000 tokens gives how many next-token examples? A: 1 · B: 500 · C: 999 · D: 1,000,000
Answer
C: 999. Every token after the first is the answer to the tokens before it.
Interview questions
- What is a language model? A model that assigns probabilities to the next token given the previous tokens; generation repeats predict → sample → append.
- Why did n-gram models give way to neural language models? Sparsity: most contexts are never seen, and counts can't generalize; embeddings let similar contexts share knowledge, and attention handles long contexts.
- What does "self-supervised" mean for LLMs? The labels (next tokens) come from the raw text itself; no human labelling.
- What loss do LLMs train with? Cross-entropy: the average of −log(probability of the correct next token).
- What is autoregressive generation? Generating one token at a time, feeding each output back as input.
- What is a base model? A pretrained model that only continues text; instruction tuning and RLHF turn it into an assistant.
- Does an LLM look facts up? No; it predicts likely text from patterns in its weights, unless given tools or retrieved documents.
Summary
- A language model predicts the next token; generation = predict, append, repeat.
- A counting (bigram) model works but greedy decoding loops forever.
- More context makes counting sparse and it can't generalize.
- A Transformer outputs a probability distribution over the vocabulary (logits → softmax).
- Self-supervised: the text is its own answer key; loss = −log(p) (cross-entropy).
- GPT-3: 175 billion parameters, 300 billion tokens, 96 layers, 2,048-token context.
- Next episode: temperature, top-p and the context window.