SyntaxPilot
Courses/AI: Zero to Agents/Lesson 22

AI: Zero to Agents · Lesson 22 of 40 · 11 min

Reasoning models

queued

These are the notes for AI episode 22. In episode 21 you asked a model to think step by step. Reasoning models do it on their own, because they were trained to. You'll learn what thinking tokens are, test-time compute (a thinking budget), how reinforcement learning with verifiable rewards teaches a model to check its work, the history of o1 and DeepSeek-R1, reasoning effort, what thinking costs, and when to use a reasoning model instead of a quick one. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13). The models in the code are toys that show the shape of the idea.

Real-life example: two quiz contestants

On the quiz show In AI
Contestant 1 answers in 1 second, first idea a regular LLM: answers straight away
Contestant 2 works on a rough sheet: tries, checks, crosses out a reasoning model: writes thinking tokens first
Contestant 2 is right, but slower better answers on hard problems, at the cost of time and tokens

Thinking: try, check, try again

Puzzle: pick two items from the menu that cost exactly ₹85.

import random
random.seed(4)
menu = {"chai": 15, "samosa": 20, "vada pav": 25, "lassi": 40, "dosa": 60}
def idea():                          # one quick guess: any 2 items
    return random.sample(list(menu), 2)
def quick(target):                   # answers its first idea, no checking
    pick = idea()
    print(f"  answer: {pick} = {sum(menu[x] for x in pick)}")
def think(target, budget):           # budget = how many steps it may think
    for step in range(1, budget + 1):
        pick = idea()
        cost = sum(menu[x] for x in pick)
        if cost == target:           # check the idea before answering
            return print(f"  step {step}: {pick} = {cost}. Correct!")
        print(f"  step {step}: {pick} = {cost}. Wait, not {target}. Again.")
    print("  out of budget: no answer")
print("quick model, target 85:");  quick(85)
print("thinking model, target 85:"); think(85, 8)

Output:

quick model, target 85:
  answer: ['samosa', 'vada pav'] = 45
thinking model, target 85:
  step 1: ['chai', 'lassi'] = 55. Wait, not 85. Again.
  step 2: ['lassi', 'samosa'] = 60. Wait, not 85. Again.
  step 3: ['chai', 'dosa'] = 75. Wait, not 85. Again.
  step 4: ['chai', 'lassi'] = 55. Wait, not 85. Again.
  step 5: ['dosa', 'vada pav'] = 85. Correct!

Real reasoning models explore far more cleverly than random guesses, but the shape is the same: write out ideas, check them, notice mistakes, try another way, then answer.

What a reasoning model is

  • A reasoning model is an LLM trained to think before it answers.
  • It first writes thinking tokens (also called reasoning tokens): a long chain of thought with steps, checks and "wait, that's wrong".
  • Then it writes the final answer. Many apps show only the answer or a summary of the thinking.
  • It still predicts one token at a time; it just spends many more tokens, and so much more compute, per question.

What a thinking trace looks like

3 chai x 15 = 45. 2 samosas x 20 = 40.
Total = 45 + 40 = 95.
Wait. 45 + 40 is 85, not 95. Let me redo it.
Total = 85. Change = 100 - 85 = 15.
Check: 85 + 15 = 100. Yes.
Final answer: Rs 15 change

The model catches its own slip and checks backwards before answering. All of that working is tokens, and tokens cost money.

Test-time compute: think longer, solve more

import random
random.seed(2)
def solved(budget):                  # one puzzle: each idea right 1 in 10
    return any(random.random() < 0.1 for _ in range(budget))

for budget in [1, 2, 4, 8, 16, 32]:
    score = sum(solved(budget) for _ in range(1000)) / 10
    print(f"thinking budget {budget:2} steps: {score:5.1f}% solved")

Output:

thinking budget  1 steps:   9.0% solved
thinking budget  2 steps:  18.1% solved
thinking budget  4 steps:  36.0% solved
thinking budget  8 steps:  58.2% solved
thinking budget 16 steps:  79.2% solved
thinking budget 32 steps:  96.3% solved

The model didn't get bigger; it thought longer. Compute spent while answering (not while training) is test-time compute: a second way to make AI smarter.

OpenAI o1 (September 2024)

From OpenAI's post "Learning to reason with LLMs" (12 September 2024):

  • o1 uses a chain of thought before answering, and was trained with large-scale reinforcement learning to refine it: to recognise and correct its mistakes and try a different approach when one isn't working.
  • Its performance improved with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).
  • On the AIME 2024 maths exams, GPT-4o solved on average 12% of problems; o1 solved 74% with one sample per problem and 83% with a consensus (vote) of 64 samples, which is self-consistency from episode 21.

Real-life example: learning to ride a cycle

Learning to cycle Training a reasoning model
Nobody can explain balance in words nobody can hand-write millions of reasoning chains
You wobble and fall wrong answer: no reward
You stay up a bit longer: do more of that right answer: reward, so that behaviour grows
A week later you ride checking and backtracking emerge

How a model learns to check its work

Nobody can hand-write millions of perfect reasoning chains. Instead: reward only verified right answers.

import random
random.seed(0)
p_check = 0.1             # at first, it rarely checks its own work

for episode in range(1, 3001):
    checks = random.random() < p_check
    correct = random.random() < (0.9 if checks else 0.5)
    if correct:            # reward: the answer is verified as right
        # make whatever it just did a little more likely
        p_check += 0.01 * ((1 if checks else 0) - p_check)
    if episode in (1, 500, 1000, 2000, 3000):
        print(f"episode {episode:4}: checks its work {p_check:.0%}")

Output:

episode    1: checks its work 10%
episode  500: checks its work 45%
episode 1000: checks its work 86%
episode 2000: checks its work 100%
episode 3000: checks its work 100%

Nobody wrote "check your work". Checking led to more rewarded answers, so it grew. This is reinforcement learning (episode 3) with a verifiable reward: a reward computed automatically, like "is the maths answer right?" or "does the code pass its tests?".

DeepSeek-R1 (January 2025)

DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (arXiv 2501.12948, 22 January 2025; published in Nature, vol. 645, 2025):

  • Reasoning abilities can be encouraged through pure reinforcement learning, without human-written reasoning examples.
  • Behaviours such as self-reflection, verification and changing strategy emerged during training.
  • Strong results on verifiable tasks: maths, coding competitions and STEM.

The price of thinking

answer = "Change: 15 rupees."
thinking = ("3 chai at 15 is 45. 2 samosas at 20 is 40. Total 45 + 40 = 85. "
            "Wait, let me check the total again: 45 + 40 = 85. Yes. "
            "Paid 100, so the change is 100 - 85 = 15.")

def tokens(text):            # rough rule: about 1 token per word
    return len(text.split())

quick = tokens(answer)
thought = tokens(thinking)
reasoning = thought + quick                 # thinking + the answer
print("quick model    :", quick, "output tokens")
print("reasoning model:", reasoning, "output tokens,", thought, "thinking")
print("cost per question:", round(reasoning / quick), "times more")

Output:

quick model    : 3 output tokens
reasoning model: 45 output tokens, 42 thinking
cost per question: 15 times more

From OpenAI's API guide on reasoning models: reasoning tokens are not visible via the API, they still occupy space in the context window, and they are billed as output tokens. More tokens also means a longer wait.

Out of budget

Same puzzle with think(85, 4):

quick model, target 85:
  answer: ['samosa', 'vada pav'] = 45
thinking model, budget 4:
  step 1: ['chai', 'lassi'] = 55. Wait, not 85. Again.
  step 2: ['lassi', 'samosa'] = 60. Wait, not 85. Again.
  step 3: ['chai', 'dosa'] = 75. Wait, not 85. Again.
  step 4: ['chai', 'lassi'] = 55. Wait, not 85. Again.
  out of budget: no answer

OpenAI's guide says that if the output token limit is reached while the model is still reasoning, the response has status incomplete, possibly with no visible output, and you still pay for the input and reasoning tokens. It recommends reserving at least 25,000 tokens for reasoning and outputs when you start, then adjusting.

Chain-of-thought prompting vs a reasoning model

Chain-of-thought prompting (episode 21) Reasoning model
works on any model a specially trained model
you ask: "think step by step" it thinks on its own
steps appear in the visible reply thinking is often hidden
quality depends on your prompt learned to check and backtrack
no special training trained with RL on rewards

Reasoning effort: a dial

Effort Trade-off
low faster, fewer tokens
medium a common default
high more complete thinking, better answers on hard problems

Many APIs let you set how much the model thinks per request; the exact levels depend on the model. It's the thinking budget as a setting.

How to prompt a reasoning model

From OpenAI's reasoning guide:

  1. A clear goal: what does "done" look like?
  2. Strong constraints: budget, rules, what to avoid.
  3. An explicit output contract: the exact format of the answer.
  4. Don't prescribe every intermediate step; it plans its own.
  5. Treat reasoning effort as a tuning knob, not the main way to fix quality. Fix the prompt first.

Quick model or reasoning model?

Quick model Reasoning model
rewrite an email politely a tricky maths or logic problem
summarise a paragraph debug a hard piece of code
translate plan a multi-step task
label 10,000 reviews analyse a long contract
a live chat reply an agent deciding what to do
simple, high volume, needs speed multi-step, hard, worth the wait

Glossary

Term Meaning Example from this episode
Reasoning model an LLM trained to think before answering o1, DeepSeek-R1
Thinking (reasoning) tokens tokens of the model's working, before the answer "Wait, not 85. Again."
Chain of thought written-out reasoning steps 45, 40, 85, 15
Backtracking dropping an idea that failed and trying another steps 1-4
Verifier / check a test of whether an answer is right cost == target
Thinking budget how many steps or tokens the model may spend budget=8
Test-time compute compute spent while answering 1 → 32 steps: 9% → 96%
Train-time compute compute spent while training more RL made o1 better
Reinforcement learning (RL) learning from rewards for actions checking grows to 100%
Verifiable reward a reward computed automatically from a checkable answer right maths answer = reward
Reasoning effort an API setting for how much the model thinks low / medium / high
Output tokens tokens the model generates (billed) 45 for the reasoning model
Latency how long you wait for the answer thinking makes it longer

Common mistakes

Code Exact error Why Fix
random.sample(list(menu), 6) with 5 items ValueError: Sample larger than population or is negative asked for more items than exist keep the sample size ≤ len(menu)
think(85) TypeError: think() missing 1 required positional argument: 'budget' every thinking loop needs a limit always pass a budget
round(reasoning / tokens("")) ZeroDivisionError: division by zero an empty answer has 0 tokens check the answer isn't empty first

Mini project: a budgeted solver with a cost report

Extend think() to return how many steps it used. Run it on 1,000 random targets from the set of all possible two-item totals, with budgets 1, 4 and 16. For each budget print: percent solved, average steps used, and "cost" = average steps × 10 tokens per step. Which budget would you choose if each token costs money? Write one sentence explaining your choice.

Practice

  1. Change the target to 100. How many steps does the thinking model need now?
  2. Make each idea right 1 in 5 (0.2) and redraw the budget table.
  3. In the RL code, make checking only slightly better: right 60% of the time when it checks (vs 50%). What happens?
  4. List three tasks from your week. Quick model or reasoning model?
Solutions

1. Change both 85s to 100:

quick model, target 100:
  answer: ['samosa', 'vada pav'] = 45
thinking model, target 100:
  step 1: ['chai', 'lassi'] = 55. Wait, not 100. Again.
  step 2: ['lassi', 'samosa'] = 60. Wait, not 100. Again.
  step 3: ['chai', 'dosa'] = 75. Wait, not 100. Again.
  step 4: ['chai', 'lassi'] = 55. Wait, not 100. Again.
  step 5: ['dosa', 'vada pav'] = 85. Wait, not 100. Again.
  step 6: ['chai', 'samosa'] = 35. Wait, not 100. Again.
  step 7: ['dosa', 'vada pav'] = 85. Wait, not 100. Again.
  step 8: ['vada pav', 'samosa'] = 45. Wait, not 100. Again.
  out of budget: no answer

Only one pair (lassi + dosa) makes 100, and this run never tried it within 8 steps. Each idea is right only 1 time in 10 here (1 good pair out of 10 possible pairs). With think(100, 16) this run still runs out of budget; with think(100, 32) it finds ['lassi', 'dosa'] = 100 at step 23. A bigger thinking budget is the fix, exactly as the budget table predicts.

2. Change 0.1 to 0.2:

thinking budget  1 steps:  19.5% solved
thinking budget  2 steps:  33.9% solved
thinking budget  4 steps:  59.8% solved
thinking budget  8 steps:  83.3% solved
thinking budget 16 steps:  97.6% solved
thinking budget 32 steps: 100.0% solved

3. Change 0.9 to 0.6:

episode    1: checks its work 10%
episode  500: checks its work 18%
episode 1000: checks its work 18%
episode 2000: checks its work 26%
episode 3000: checks its work 56%

Checking still grows, but far more slowly and noisily (56% after 3,000 episodes instead of 100% by 2,000), because the reward difference is small.

4. Example: replying to a friend (quick), summarising a chapter (quick), solving a hard DSA problem (reasoning).

Quick quiz 1

Same model, same weights, but it solves more puzzles. Why? A: trained on more data · B: more parameters · C: it spends more compute thinking per question · D: the temperature went up

Answer

C. That's test-time compute: more thinking tokens, more checks.

Quick quiz 2

Label 1 million short reviews as positive/negative by tonight. Which model? A: a reasoning model, high effort · B: a quick model · C: a reasoning model, ask 64 times · D: no model can do this

Answer

B. Simple, high-volume work: a quick model is faster and far cheaper.

Interview questions

  1. What is a reasoning model, and how is it different from a regular LLM? It generates an internal chain of thought (reasoning tokens) before the final answer, and is trained (typically with RL) to use that thinking to check and correct itself.
  2. What is test-time compute? Compute spent at inference, e.g. longer reasoning, more samples or voting; accuracy on hard problems rises with it.
  3. How are reasoning models trained? Largely with reinforcement learning on tasks with verifiable rewards (maths answers, unit tests), which rewards correct final answers and lets behaviours like self-verification emerge (o1, DeepSeek-R1).
  4. What are the costs of reasoning models? More output tokens (reasoning tokens are billed as output), higher latency, and context-window space used by thinking.
  5. What does a reasoning-effort setting do? It trades speed and tokens for more thorough thinking.
  6. When would you not use a reasoning model? Simple, high-volume or latency-sensitive tasks: classification, rewriting, short chat replies.

Summary

  • A reasoning model writes thinking tokens, checks ideas and tries again before answering.
  • Test-time compute: thinking longer solves more.
  • RL with verifiable rewards taught models like o1 (Sep 2024) and DeepSeek-R1 (Jan 2025) to check their work.
  • Thinking is billed as output tokens and fills the context window; it also adds wait time.
  • Use reasoning effort to set how much it thinks, and pick quick vs reasoning by the task.
  • Next episode: calling an LLM from code.