These are the notes for AI episode 22. In episode 21 you asked a model to think step by step. Reasoning models do it on their own, because they were trained to. You'll learn what thinking tokens are, test-time compute (a thinking budget), how reinforcement learning with verifiable rewards teaches a model to check its work, the history of o1 and DeepSeek-R1, reasoning effort, what thinking costs, and when to use a reasoning model instead of a quick one. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13). The models in the code are toys that show the shape of the idea.
Real-life example: two quiz contestants
| On the quiz show | In AI |
|---|---|
| Contestant 1 answers in 1 second, first idea | a regular LLM: answers straight away |
| Contestant 2 works on a rough sheet: tries, checks, crosses out | a reasoning model: writes thinking tokens first |
| Contestant 2 is right, but slower | better answers on hard problems, at the cost of time and tokens |
Thinking: try, check, try again
Puzzle: pick two items from the menu that cost exactly ₹85.
import random
random.seed(4)
menu = {"chai": 15, "samosa": 20, "vada pav": 25, "lassi": 40, "dosa": 60}
def idea(): # one quick guess: any 2 items
return random.sample(list(menu), 2)
def quick(target): # answers its first idea, no checking
pick = idea()
print(f" answer: {pick} = {sum(menu[x] for x in pick)}")
def think(target, budget): # budget = how many steps it may think
for step in range(1, budget + 1):
pick = idea()
cost = sum(menu[x] for x in pick)
if cost == target: # check the idea before answering
return print(f" step {step}: {pick} = {cost}. Correct!")
print(f" step {step}: {pick} = {cost}. Wait, not {target}. Again.")
print(" out of budget: no answer")
print("quick model, target 85:"); quick(85)
print("thinking model, target 85:"); think(85, 8)
Output:
quick model, target 85:
answer: ['samosa', 'vada pav'] = 45
thinking model, target 85:
step 1: ['chai', 'lassi'] = 55. Wait, not 85. Again.
step 2: ['lassi', 'samosa'] = 60. Wait, not 85. Again.
step 3: ['chai', 'dosa'] = 75. Wait, not 85. Again.
step 4: ['chai', 'lassi'] = 55. Wait, not 85. Again.
step 5: ['dosa', 'vada pav'] = 85. Correct!
Real reasoning models explore far more cleverly than random guesses, but the shape is the same: write out ideas, check them, notice mistakes, try another way, then answer.
What a reasoning model is
- A reasoning model is an LLM trained to think before it answers.
- It first writes thinking tokens (also called reasoning tokens): a long chain of thought with steps, checks and "wait, that's wrong".
- Then it writes the final answer. Many apps show only the answer or a summary of the thinking.
- It still predicts one token at a time; it just spends many more tokens, and so much more compute, per question.
What a thinking trace looks like
3 chai x 15 = 45. 2 samosas x 20 = 40.
Total = 45 + 40 = 95.
Wait. 45 + 40 is 85, not 95. Let me redo it.
Total = 85. Change = 100 - 85 = 15.
Check: 85 + 15 = 100. Yes.
Final answer: Rs 15 change
The model catches its own slip and checks backwards before answering. All of that working is tokens, and tokens cost money.
Test-time compute: think longer, solve more
import random
random.seed(2)
def solved(budget): # one puzzle: each idea right 1 in 10
return any(random.random() < 0.1 for _ in range(budget))
for budget in [1, 2, 4, 8, 16, 32]:
score = sum(solved(budget) for _ in range(1000)) / 10
print(f"thinking budget {budget:2} steps: {score:5.1f}% solved")
Output:
thinking budget 1 steps: 9.0% solved
thinking budget 2 steps: 18.1% solved
thinking budget 4 steps: 36.0% solved
thinking budget 8 steps: 58.2% solved
thinking budget 16 steps: 79.2% solved
thinking budget 32 steps: 96.3% solved
The model didn't get bigger; it thought longer. Compute spent while answering (not while training) is test-time compute: a second way to make AI smarter.
OpenAI o1 (September 2024)
From OpenAI's post "Learning to reason with LLMs" (12 September 2024):
- o1 uses a chain of thought before answering, and was trained with large-scale reinforcement learning to refine it: to recognise and correct its mistakes and try a different approach when one isn't working.
- Its performance improved with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).
- On the AIME 2024 maths exams, GPT-4o solved on average 12% of problems; o1 solved 74% with one sample per problem and 83% with a consensus (vote) of 64 samples, which is self-consistency from episode 21.
Real-life example: learning to ride a cycle
| Learning to cycle | Training a reasoning model |
|---|---|
| Nobody can explain balance in words | nobody can hand-write millions of reasoning chains |
| You wobble and fall | wrong answer: no reward |
| You stay up a bit longer: do more of that | right answer: reward, so that behaviour grows |
| A week later you ride | checking and backtracking emerge |
How a model learns to check its work
Nobody can hand-write millions of perfect reasoning chains. Instead: reward only verified right answers.
import random
random.seed(0)
p_check = 0.1 # at first, it rarely checks its own work
for episode in range(1, 3001):
checks = random.random() < p_check
correct = random.random() < (0.9 if checks else 0.5)
if correct: # reward: the answer is verified as right
# make whatever it just did a little more likely
p_check += 0.01 * ((1 if checks else 0) - p_check)
if episode in (1, 500, 1000, 2000, 3000):
print(f"episode {episode:4}: checks its work {p_check:.0%}")
Output:
episode 1: checks its work 10%
episode 500: checks its work 45%
episode 1000: checks its work 86%
episode 2000: checks its work 100%
episode 3000: checks its work 100%
Nobody wrote "check your work". Checking led to more rewarded answers, so it grew. This is reinforcement learning (episode 3) with a verifiable reward: a reward computed automatically, like "is the maths answer right?" or "does the code pass its tests?".
DeepSeek-R1 (January 2025)
DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (arXiv 2501.12948, 22 January 2025; published in Nature, vol. 645, 2025):
- Reasoning abilities can be encouraged through pure reinforcement learning, without human-written reasoning examples.
- Behaviours such as self-reflection, verification and changing strategy emerged during training.
- Strong results on verifiable tasks: maths, coding competitions and STEM.
The price of thinking
answer = "Change: 15 rupees."
thinking = ("3 chai at 15 is 45. 2 samosas at 20 is 40. Total 45 + 40 = 85. "
"Wait, let me check the total again: 45 + 40 = 85. Yes. "
"Paid 100, so the change is 100 - 85 = 15.")
def tokens(text): # rough rule: about 1 token per word
return len(text.split())
quick = tokens(answer)
thought = tokens(thinking)
reasoning = thought + quick # thinking + the answer
print("quick model :", quick, "output tokens")
print("reasoning model:", reasoning, "output tokens,", thought, "thinking")
print("cost per question:", round(reasoning / quick), "times more")
Output:
quick model : 3 output tokens
reasoning model: 45 output tokens, 42 thinking
cost per question: 15 times more
From OpenAI's API guide on reasoning models: reasoning tokens are not visible via the API, they still occupy space in the context window, and they are billed as output tokens. More tokens also means a longer wait.
Out of budget
Same puzzle with think(85, 4):
quick model, target 85:
answer: ['samosa', 'vada pav'] = 45
thinking model, budget 4:
step 1: ['chai', 'lassi'] = 55. Wait, not 85. Again.
step 2: ['lassi', 'samosa'] = 60. Wait, not 85. Again.
step 3: ['chai', 'dosa'] = 75. Wait, not 85. Again.
step 4: ['chai', 'lassi'] = 55. Wait, not 85. Again.
out of budget: no answer
OpenAI's guide says that if the output token limit is reached while the model is still reasoning, the response has status incomplete, possibly with no visible output, and you still pay for the input and reasoning tokens. It recommends reserving at least 25,000 tokens for reasoning and outputs when you start, then adjusting.
Chain-of-thought prompting vs a reasoning model
| Chain-of-thought prompting (episode 21) | Reasoning model |
|---|---|
| works on any model | a specially trained model |
| you ask: "think step by step" | it thinks on its own |
| steps appear in the visible reply | thinking is often hidden |
| quality depends on your prompt | learned to check and backtrack |
| no special training | trained with RL on rewards |
Reasoning effort: a dial
| Effort | Trade-off |
|---|---|
| low | faster, fewer tokens |
| medium | a common default |
| high | more complete thinking, better answers on hard problems |
Many APIs let you set how much the model thinks per request; the exact levels depend on the model. It's the thinking budget as a setting.
How to prompt a reasoning model
From OpenAI's reasoning guide:
- A clear goal: what does "done" look like?
- Strong constraints: budget, rules, what to avoid.
- An explicit output contract: the exact format of the answer.
- Don't prescribe every intermediate step; it plans its own.
- Treat reasoning effort as a tuning knob, not the main way to fix quality. Fix the prompt first.
Quick model or reasoning model?
| Quick model | Reasoning model |
|---|---|
| rewrite an email politely | a tricky maths or logic problem |
| summarise a paragraph | debug a hard piece of code |
| translate | plan a multi-step task |
| label 10,000 reviews | analyse a long contract |
| a live chat reply | an agent deciding what to do |
| simple, high volume, needs speed | multi-step, hard, worth the wait |
Glossary
| Term | Meaning | Example from this episode |
|---|---|---|
| Reasoning model | an LLM trained to think before answering | o1, DeepSeek-R1 |
| Thinking (reasoning) tokens | tokens of the model's working, before the answer | "Wait, not 85. Again." |
| Chain of thought | written-out reasoning steps | 45, 40, 85, 15 |
| Backtracking | dropping an idea that failed and trying another | steps 1-4 |
| Verifier / check | a test of whether an answer is right | cost == target |
| Thinking budget | how many steps or tokens the model may spend | budget=8 |
| Test-time compute | compute spent while answering | 1 → 32 steps: 9% → 96% |
| Train-time compute | compute spent while training | more RL made o1 better |
| Reinforcement learning (RL) | learning from rewards for actions | checking grows to 100% |
| Verifiable reward | a reward computed automatically from a checkable answer | right maths answer = reward |
| Reasoning effort | an API setting for how much the model thinks | low / medium / high |
| Output tokens | tokens the model generates (billed) | 45 for the reasoning model |
| Latency | how long you wait for the answer | thinking makes it longer |
Common mistakes
| Code | Exact error | Why | Fix |
|---|---|---|---|
random.sample(list(menu), 6) with 5 items |
ValueError: Sample larger than population or is negative |
asked for more items than exist | keep the sample size ≤ len(menu) |
think(85) |
TypeError: think() missing 1 required positional argument: 'budget' |
every thinking loop needs a limit | always pass a budget |
round(reasoning / tokens("")) |
ZeroDivisionError: division by zero |
an empty answer has 0 tokens | check the answer isn't empty first |
Mini project: a budgeted solver with a cost report
Extend think() to return how many steps it used. Run it on 1,000 random targets from the set of all possible two-item totals, with budgets 1, 4 and 16. For each budget print: percent solved, average steps used, and "cost" = average steps × 10 tokens per step. Which budget would you choose if each token costs money? Write one sentence explaining your choice.
Practice
- Change the target to 100. How many steps does the thinking model need now?
- Make each idea right 1 in 5 (
0.2) and redraw the budget table. - In the RL code, make checking only slightly better: right 60% of the time when it checks (vs 50%). What happens?
- List three tasks from your week. Quick model or reasoning model?
Solutions
1. Change both 85s to 100:
quick model, target 100:
answer: ['samosa', 'vada pav'] = 45
thinking model, target 100:
step 1: ['chai', 'lassi'] = 55. Wait, not 100. Again.
step 2: ['lassi', 'samosa'] = 60. Wait, not 100. Again.
step 3: ['chai', 'dosa'] = 75. Wait, not 100. Again.
step 4: ['chai', 'lassi'] = 55. Wait, not 100. Again.
step 5: ['dosa', 'vada pav'] = 85. Wait, not 100. Again.
step 6: ['chai', 'samosa'] = 35. Wait, not 100. Again.
step 7: ['dosa', 'vada pav'] = 85. Wait, not 100. Again.
step 8: ['vada pav', 'samosa'] = 45. Wait, not 100. Again.
out of budget: no answer
Only one pair (lassi + dosa) makes 100, and this run never tried it within 8 steps. Each idea is right only 1 time in 10 here (1 good pair out of 10 possible pairs). With think(100, 16) this run still runs out of budget; with think(100, 32) it finds ['lassi', 'dosa'] = 100 at step 23. A bigger thinking budget is the fix, exactly as the budget table predicts.
2. Change 0.1 to 0.2:
thinking budget 1 steps: 19.5% solved
thinking budget 2 steps: 33.9% solved
thinking budget 4 steps: 59.8% solved
thinking budget 8 steps: 83.3% solved
thinking budget 16 steps: 97.6% solved
thinking budget 32 steps: 100.0% solved
3. Change 0.9 to 0.6:
episode 1: checks its work 10%
episode 500: checks its work 18%
episode 1000: checks its work 18%
episode 2000: checks its work 26%
episode 3000: checks its work 56%
Checking still grows, but far more slowly and noisily (56% after 3,000 episodes instead of 100% by 2,000), because the reward difference is small.
4. Example: replying to a friend (quick), summarising a chapter (quick), solving a hard DSA problem (reasoning).
Quick quiz 1
Same model, same weights, but it solves more puzzles. Why? A: trained on more data · B: more parameters · C: it spends more compute thinking per question · D: the temperature went up
Answer
C. That's test-time compute: more thinking tokens, more checks.
Quick quiz 2
Label 1 million short reviews as positive/negative by tonight. Which model? A: a reasoning model, high effort · B: a quick model · C: a reasoning model, ask 64 times · D: no model can do this
Answer
B. Simple, high-volume work: a quick model is faster and far cheaper.
Interview questions
- What is a reasoning model, and how is it different from a regular LLM? It generates an internal chain of thought (reasoning tokens) before the final answer, and is trained (typically with RL) to use that thinking to check and correct itself.
- What is test-time compute? Compute spent at inference, e.g. longer reasoning, more samples or voting; accuracy on hard problems rises with it.
- How are reasoning models trained? Largely with reinforcement learning on tasks with verifiable rewards (maths answers, unit tests), which rewards correct final answers and lets behaviours like self-verification emerge (o1, DeepSeek-R1).
- What are the costs of reasoning models? More output tokens (reasoning tokens are billed as output), higher latency, and context-window space used by thinking.
- What does a reasoning-effort setting do? It trades speed and tokens for more thorough thinking.
- When would you not use a reasoning model? Simple, high-volume or latency-sensitive tasks: classification, rewriting, short chat replies.
Summary
- A reasoning model writes thinking tokens, checks ideas and tries again before answering.
- Test-time compute: thinking longer solves more.
- RL with verifiable rewards taught models like o1 (Sep 2024) and DeepSeek-R1 (Jan 2025) to check their work.
- Thinking is billed as output tokens and fills the context window; it also adds wait time.
- Use reasoning effort to set how much it thinks, and pick quick vs reasoning by the task.
- Next episode: calling an LLM from code.