These are the notes for AI episode 17. Last episode you learned the knobs that control what a model writes. But a model fresh out of training is not an assistant: ask it a question and it may ask you another one. Here you invent the steps that turn a base model into a helpful assistant: fine-tuning, supervised fine-tuning (SFT) with a chat template, human preferences, a reward model, reinforcement learning from human feedback (RLHF), reward hacking and the KL penalty, and the simpler DPO. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).
Real-life example: the bookworm at the enquiry counter
| At the railway counter | In an LLM |
|---|---|
| Someone who has read every book in a huge library | a base model after pretraining on huge amounts of text |
| A passenger asks "When does the Rajdhani leave?" | a user's prompt |
| They read out the next line of the timetable: "When does the Shatabdi leave?" | the base model continues the text instead of answering |
| On-the-job training: how to answer, politely | fine-tuning and RLHF |
Attempt: ask a base model a question
A tiny language model like the one from episode 15: it learns which word follows each pair of words. Its training text is a page of quiz questions, plus one answer.
import random
from collections import defaultdict
random.seed(5)
web = """Q: what is the capital of india ? Q: what is the capital of france ?
Q: what is the largest planet ? Q: how do i make chai ? Q: what is
the capital of japan ? the capital of india is delhi ."""
model = defaultdict(list) # pretraining: last 2 words -> next
w = web.split()
for i in range(len(w) - 2):
model[(w[i], w[i + 1])].append(w[i + 2])
out = "what is the capital of india ?".split()
for _ in range(10): # generate: predict, append, repeat
out.append(random.choice(model.get(tuple(out[-2:]), ["<end>"])))
if out[-1] == "<end>": break
print(" ".join(out))
Output:
what is the capital of india ? Q: how do i make chai ? Q: how do
It didn't answer: it asked another question. Learning from a big pile of text is pretraining, and the result is a base model. A base model only learned to continue text the way its training text continues, and on a quiz page a question is followed by another question.
This happens with big models too. In OpenAI's 2022 InstructGPT paper (Ouyang et al., arXiv 2203.02155), GPT-3 was given a French instruction to write a short story about a frog, and replied with more instructions instead of the story. The paper opens: "Making language models bigger does not inherently make them better at following a user's intent." Getting a model to do what people actually want is called alignment.
Fine-tuning and supervised fine-tuning (SFT)
- Start from the pretrained base model (it already knows a lot).
- Keep training it, the same way, on a small new dataset: that is fine-tuning.
- The dataset is demonstrations: a prompt plus the ideal answer, written by a person.
- Mark who is speaking with a fixed pattern (
User: ... Assistant: ...): a chat template. - Learning from demonstrations is supervised fine-tuning (SFT): "supervised" because the right answer is given.
import random
from collections import defaultdict
random.seed(5)
web = """Q: what is the capital of india ? Q: what is the capital of france ?
Q: what is the largest planet ? Q: how do i make chai ? Q: what is"""
demos = """User: what is the capital of india ? Assistant: delhi . <end>
User: how do i make chai ? Assistant: boil tea leaves with milk . <end>"""
model = defaultdict(list)
for text in [web, demos]: # pretrain, then fine-tune
w = text.split()
for i in range(len(w) - 2):
model[(w[i], w[i + 1])].append(w[i + 2])
for q in ["how do i make chai ?", "what is the capital of india ?"]:
out = ("User: " + q + " Assistant:").split()
while out[-1] != "<end>" and len(out) < 30:
out.append(random.choice(model.get(tuple(out[-2:]), ["<end>"])))
print(q, "->", " ".join(out[len(q.split()) + 2:]))
Output:
how do i make chai ? -> boil tea leaves with milk . <end>
what is the capital of india ? -> delhi . <end>
It answers now, and stops at the end token <end>. Fine-tuning taught the format and the job (when you see Assistant:, answer, then stop), not much new knowledge.
Honest note: our toy only looks at the last 2 words, so after
? Assistant:it can't tell the two questions apart; withrandom.seed(5)both happened to land on the right answer. A real transformer reads the whole prompt with attention. Practice question 1 shows the fix for the toy.
Why demonstrations are not enough
| Problem | Why it matters |
|---|---|
| Slow and costly to write | a person writes every ideal answer (InstructGPT used about 13k prompts for SFT) |
| One answer per prompt | the model copies it, but never learns which answers are worse, or why |
| Judging is easier than doing | at a cooking contest the judges taste two dishes; they don't need to be chefs |
So: show people two answers and ask which is better. Those picks are human preferences.
The reward model
from collections import defaultdict
# people compared two replies to "how do i make chai ?" and picked one
prefs = [("boil tea leaves with milk and sugar .", "figure it out yourself ."),
("sure ! boil water , add tea and milk .", "chai is a drink ."),
("add ginger , boil with milk and tea .", "no .")]
weight = defaultdict(float) # the reward model: a score per word
def reward(reply):
return sum(weight[w] for w in reply.split())
for _ in range(5): # learn: better reply must score higher
for better, worse in prefs:
if reward(better) <= reward(worse):
for w in better.split(): weight[w] += 0.1
for w in worse.split(): weight[w] -= 0.1
for reply in ["boil milk and tea leaves .", "figure it out .", "chai ."]:
print(f"{reward(reply):5.1f} {reply}")
Output:
0.5 boil milk and tea leaves .
-0.3 figure it out .
0.0 chai .
A model that gives any reply a score, trained from comparisons, is a reward model. It scored three replies it had never seen in a sensible order. Now we have an automatic judge that can score millions of replies without a person.
Reinforcement learning from human feedback (RLHF)
import math, random
random.seed(1)
replies = ["boil tea leaves with milk .", "figure it out .", "chai ."]
reward = [0.5, -0.3, 0.0] # from our reward model
score = [0.0, 0.0, 0.0] # the model's preference for each reply
def probs():
e = [math.exp(s) for s in score]
return [round(x / sum(e), 2) for x in e]
print("before:", probs())
for step in range(300): # reinforcement learning
i = random.choices(range(3), weights=probs())[0] # model answers
score[i] += 0.5 * reward[i] # reward nudges it
print("after: ", probs())
Output:
before: [0.33, 0.33, 0.33]
after: [1.0, 0.0, 0.0]
- Reinforcement learning (RL): learn by trying and getting a reward, instead of being shown the right answer.
- Policy: the model being trained by RL (it decides what to say).
- RLHF: RL where the reward comes from a reward model trained on human preferences.
How InstructGPT was trained (Ouyang et al., 2022)
GPT-3 (base) ──► 1. SFT ──► 2. Reward model ──► 3. RL with PPO ──► InstructGPT
written ranked answers reward model
answers as the judge
| Step | Data | What it learns |
|---|---|---|
| 1. Supervised fine-tuning | about 13k prompts with answers written by labelers | the job and the format |
| 2. Reward model | 33k prompts, labelers ranked several model answers | to score answers like people do |
| 3. Reinforcement learning (PPO, proximal policy optimization) | 31k prompts | to write answers the reward model scores highly |
- About 40 contractors (labelers) wrote and ranked the answers.
- People preferred outputs of the 1.3B-parameter InstructGPT over the 175B GPT-3, a model over 100 times bigger.
- ChatGPT (OpenAI, 30 November 2022) was described as "a sibling model to InstructGPT".
Reward hacking
from collections import defaultdict
prefs = [("boil tea leaves with milk and sugar .", "figure it out yourself ."),
("sure ! boil water , add tea and milk .", "chai is a drink ."),
("add ginger , boil with milk and tea .", "no .")]
weight = defaultdict(float)
def reward(reply):
return sum(weight[w] for w in reply.split())
for _ in range(5):
for better, worse in prefs:
if reward(better) <= reward(worse):
for w in better.split(): weight[w] += 0.1
for w in worse.split(): weight[w] -= 0.1
for reply in ["boil tea leaves with milk and sugar .",
"milk milk milk milk milk milk milk milk"]:
print(f"{reward(reply):5.1f} {reply}")
Output:
0.7 boil tea leaves with milk and sugar .
0.8 milk milk milk milk milk milk milk milk
Nonsense beats the real recipe. The reward model is only a rough copy of what people like: it learned "milk" is a good word, so more milk means more points. RL hunts for the highest score and finds this hole: reward hacking.
The KL penalty: a kite on a string
| Flying a kite | In RLHF |
|---|---|
| The wind pulls it higher | the reward pulls toward higher scores |
| The string keeps it near your hand | the KL penalty: a cost for drifting from the fine-tuned model |
| It flies high but never flies away | better replies that stay sensible |
import math
reward = [0.7, 0.8] # recipe vs spam: spam wins!
ref = [0.9, 0.1] # the fine-tuned model before RL
def rlhf(beta, steps=3000):
score = [math.log(x) for x in ref]
for _ in range(steps):
e = [math.exp(s) for s in score]
p = [x / sum(e) for x in e]
# reward, minus a penalty for drifting away from the old model
r = [reward[j] - beta * math.log(p[j] / ref[j]) for j in range(2)]
avg = sum(p[j] * r[j] for j in range(2))
for j in range(2):
score[j] += 1.0 * p[j] * (r[j] - avg) # nudge toward better
return [round(x, 2) for x in p]
print("no leash: ", rlhf(beta=0.0))
print("with leash:", rlhf(beta=0.5))
Output:
no leash: [0.0, 1.0]
with leash: [0.88, 0.12]
log(p / ref) measures how far the model has drifted from the reference model; on average it is the KL divergence, a measure of how different two probability distributions are. beta is the strength of the leash. With no leash the model always says "milk milk milk"; with beta = 0.5 it keeps the recipe 88% of the time.
DPO: skip the separate judge
| RLHF | DPO (direct preference optimization) | |
|---|---|---|
| Stages | SFT, reward model, RL | SFT, then one training stage on the preferences |
| Separate reward model | yes | no |
| RL loop (sampling during training) | yes | no |
| Data | pairs: answer A is better than answer B | the same pairs |
DPO was proposed by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn (arXiv 2305.18290, May 2023): "Your Language Model is Secretly a Reward Model".
Glossary
| Term | Meaning | Example from this episode |
|---|---|---|
| Pretraining | training on a huge pile of text to predict the next token | the quiz-page text |
| Base model | the model right after pretraining; it continues text | asks another question back |
| Alignment | making a model do what people actually want | answering, politely |
| Fine-tuning | continuing to train an already trained model on new data | training on the demonstrations |
| Demonstration | a prompt plus an ideal answer written by a person | "how do i make chai ?" → "boil tea leaves with milk ." |
| Supervised fine-tuning (SFT) | fine-tuning on demonstrations (the right answer is given) | step 1 of InstructGPT |
| Chat template | a fixed pattern marking who speaks | User: ... Assistant: ... |
| End token | a special token that means "stop" | <end> |
| Human preference | a person's pick of the better of two answers | recipe beats "figure it out yourself" |
| Labeler | a person hired to write or rank answers | about 40 for InstructGPT |
| Reward model | a model that scores replies, trained on preferences | our score-per-word model |
| Reinforcement learning (RL) | learning by trying and getting a reward | the 300-step loop |
| Policy | the model being trained by RL | our three-reply model |
| RLHF | reinforcement learning from human feedback | SFT → reward model → RL |
| PPO | proximal policy optimization, the RL algorithm InstructGPT used | step 3 |
| Reward hacking | the policy gaming the reward model | "milk milk milk" scores 0.8 |
| KL penalty | a cost for drifting from the starting model (beta sets its strength) | the kite string |
| KL divergence | a measure of how different two probability distributions are | average of log(p / ref) |
| DPO | direct preference optimization: train on preference pairs directly | no separate reward model |
Common mistakes
| Code | Exact error | Why | Fix |
|---|---|---|---|
random.choice(model[("hello", "there")]) with a defaultdict(list) |
IndexError: Cannot choose from an empty sequence |
that word pair never appeared in training | model.get(key, ["<end>"]): stop on unseen text |
math.log(p / ref) with p = 0 |
ValueError: math domain error |
the log of zero is undefined | never let a probability hit exactly 0 |
math.exp(1000) |
OverflowError: math range error |
scores grew too large: too big a step | use a smaller step size |
Mini project: a tiny instruction-following bot
Write 5 demonstrations in the chat template (User: ... Assistant: ... <end>) about your city (famous food, a monument, the best time to visit). Fine-tune the 4-word model from practice question 1 on them and check every question gets its own answer. Then write 3 preference pairs and score 3 new replies with the reward model.
Practice
- Add a third demonstration ("what is the largest planet ?" → "jupiter .") to the SFT demo. What goes wrong with the 2-word model, and how do you fix it?
- Add your own preference pair to the reward model and score a reply it has never seen.
- In the leash demo, try
beta = 0.1andbeta = 2.0. What changes? - Find a reply that scores even higher than "milk milk milk" on the reward model.
Solutions
1. With three demonstrations, ? Assistant: is followed by three different answers, so the 2-word model mixes them up. With random.seed(5) it answered:
how do i make chai ? -> jupiter . <end>
what is the capital of india ? -> jupiter . <end>
what is the largest planet ? -> delhi . <end>
Look at the last 4 words instead:
import random
from collections import defaultdict
random.seed(5)
web = """Q: what is the capital of india ? Q: what is the capital of france ?
Q: what is the largest planet ? Q: how do i make chai ? Q: what is"""
demos = """User: what is the capital of india ? Assistant: delhi . <end>
User: how do i make chai ? Assistant: boil tea leaves with milk . <end>
User: what is the largest planet ? Assistant: jupiter . <end>"""
model = defaultdict(list)
for text in [web, demos]: # pretrain, then fine-tune
w = text.split()
for i in range(len(w) - 4): # now: last 4 words -> next
model[tuple(w[i:i + 4])].append(w[i + 4])
for q in ["how do i make chai ?", "what is the capital of india ?",
"what is the largest planet ?"]:
out = ("User: " + q + " Assistant:").split()
while out[-1] != "<end>" and len(out) < 30:
out.append(random.choice(model.get(tuple(out[-4:]), ["<end>"])))
print(q, "->", " ".join(out[len(q.split()) + 2:]))
how do i make chai ? -> boil tea leaves with milk . <end>
what is the capital of india ? -> delhi . <end>
what is the largest planet ? -> jupiter . <end>
More context lets the model tell the questions apart, which is exactly why transformers look at the whole prompt.
2. Add ("boil milk , add tea and cardamom .", "google it .") to prefs and score new replies:
for reply in ["boil milk and tea leaves .", "google it .", "add cardamom ."]:
print(f"{reward(reply):5.1f} {reply}")
0.5 boil milk and tea leaves .
-0.1 google it .
0.0 add cardamom .
"google it ." is now judged a poor reply.
3.
beta 0.1: [0.77, 0.23]
beta 2.0: [0.9, 0.1]
A weak leash (0.1) lets spam grow to 23%; a strong leash (2.0) keeps the model exactly where it started (0.9, 0.1), so it also can't improve. beta is a trade-off.
4. Repeat every positive word:
for reply in ["boil tea leaves with milk and sugar .",
"milk milk milk milk milk milk milk milk",
"boil boil boil tea tea tea milk milk milk"]:
print(f"{reward(reply):5.1f} {reply}")
0.7 boil tea leaves with milk and sugar .
0.8 milk milk milk milk milk milk milk milk
0.9 boil boil boil tea tea tea milk milk milk
The score keeps rising with nonsense, which is reward hacking in action.
Quick quiz 1
People see two answers and click the better one. Which step uses this data? A: pretraining · B: writing demonstrations · C: training the reward model · D: the context window
Answer
C: training the reward model. Pretraining uses raw text; SFT uses written ideal answers.
Quick quiz 2
After RL, a model starts every reply with "Great question! Great question! Great question!". What went wrong? A: reward hacking · B: too little pretraining · C: a chat template bug · D: top-p too low
Answer
A: reward hacking. The reward model likes that phrase, so RL found a way to game it. A stronger KL leash and better preference data help.
Interview questions
- Base model vs instruct/chat model? A base model is only pretrained and continues text; an instruct model is fine-tuned (SFT, then RLHF or DPO) to follow instructions in a chat template.
- What are the three stages of RLHF in InstructGPT? Supervised fine-tuning on demonstrations, a reward model trained on human rankings, and RL (PPO) against the reward model.
- Why use preferences instead of only demonstrations? Comparing is cheaper and easier than writing ideal answers, and comparisons teach what makes one answer better than another.
- What is reward hacking? The policy finds outputs that score high on the (imperfect) reward model without being good.
- Why is there a KL penalty in RLHF? It keeps the policy close to the SFT model, which limits reward hacking and keeps the language fluent.
- How is DPO different from RLHF? DPO trains directly on preference pairs in one stage, with no separate reward model and no RL sampling loop.
- Does fine-tuning add knowledge? Mostly it changes behaviour and format; most knowledge comes from pretraining.
Summary
- Pretraining gives a base model that only continues text.
- Fine-tuning keeps training it; SFT uses demonstrations in a chat template.
- Demonstrations are costly and show one answer each, so people compare answers: human preferences.
- A reward model learns to score replies; reinforcement learning pushes the policy toward high reward: RLHF (InstructGPT: SFT → RM → PPO).
- The policy can game the judge (reward hacking), so a KL penalty keeps it near where it started.
- DPO does the same job in one stage, from the same preference pairs.
- Next episode: multimodal AI: how models see images and hear audio.