These are the notes for AI episode 7. In episode 4 we trained a model by trying every possible line. Here you see why that is impossible for real AI, and invent the method almost every modern model uses to learn: gradient descent. Along the way: parameters, mean squared error, the slope (gradient), the learning rate, steps and epochs, the loss curve, local minima and batches. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real.
A model is a box of knobs
| A music mixer | In machine learning |
|---|---|
| Volume, bass and treble knobs | Parameters: the numbers a model learns (our chai line had two: start and drop) |
| "Too loud!": how bad it sounds | Loss: one number saying how wrong the model is |
| Turning knobs and listening, again and again | Training: changing the parameters to lower the loss |
Demo 1: trying everything fails
# Episode 4's training: try 41 values for each knob
values_per_knob = 41
tries_per_second = 1_000_000 # a fast computer
seconds_per_year = 60 * 60 * 24 * 365
for knobs in [2, 10, 100, 1000]:
tries = values_per_knob ** knobs
years = tries / tries_per_second / seconds_per_year
print(f"{knobs:4} knobs: {years:.1e} years")
Output:
2 knobs: 5.3e-11 years
10 knobs: 4.3e+02 years
100 knobs: 6.0e+147 years
Traceback (most recent call last):
File "main.py", line 8, in <module>
years = tries / tries_per_second / seconds_per_year
~~~~~~^~~~~~~~~~~~~~~~~~
OverflowError: integer division result too large for a float
- 2 knobs: 1,681 tries, a tiny fraction of a second.
- 10 knobs: about 430 years.
- 100 knobs: 6 × 10¹⁴⁷ years.
- 1,000 knobs: the number is too big for Python to even turn into a decimal (
OverflowError).
Every new knob multiplies the number of tries by 41. Modern AI models have millions or billions of parameters. We need a way to improve without trying everything.
The fix: walk downhill on a foggy hill
| On the foggy hill | In machine learning |
|---|---|
| Thick fog: you can't see the valley | We can't check every answer |
| Feel the ground: which way slopes down? | The slope, or gradient |
| Take one step downhill, then feel again | One step of training |
| Reach the bottom of the valley | The lowest loss |
The height of the hill is the loss; your position is the knob settings. Walking downhill on the loss is called gradient descent.
repeat:
slope = how the loss changes if I nudge the knob
knob = knob - learning_rate × slope # step AGAINST the slope = downhill
A smooth hill: mean squared error
In episode 4 the loss was the mean absolute error (the average |gap|). For walking downhill we usually square each gap: the mean squared error (MSE).
| Gap | Absolute error | Squared error |
|---|---|---|
| 1 | 1 | 1 |
| 5 | 5 | 25 |
| 10 | 10 | 100 |
- Big mistakes cost much more.
- The loss becomes a smooth bowl: steep far from the answer, gentle near the bottom. So the slope tells us which way to go and roughly how far.
Feeling the slope in code: nudge a knob by a tiny amount (0.001) and see how much the loss changes: slope = (loss(x + 0.001) - loss(x)) / 0.001. This is a numerical slope; real libraries compute exact slopes with calculus.
Demo 2: gradient descent on one knob
To keep it simple we measure temperature from 24 °C (t - 24) and fix base at the average number of cups, so there is only one knob to learn: drop.
# (temperature °C, cups sold); x = degrees above 24 °C
days = [(14, 96), (18, 88), (22, 81), (26, 72), (30, 63), (34, 57)]
base = 76.17 # average cups (at 24 °C)
def loss(drop): # mean squared error
return sum((base - drop * (t - 24) - cups) ** 2
for t, cups in days) / len(days)
drop = 0.0 # start anywhere
learning_rate = 0.002
print(f"start: drop = {drop:.3f} loss = {loss(drop):.2f}")
for step in range(1, 21):
slope = (loss(drop + 0.001) - loss(drop)) / 0.001 # feel the ground
drop = drop - learning_rate * slope # step downhill
if step % 4 == 0:
print(f"step {step:2}: drop = {drop:.3f} loss = {loss(drop):.2f}")
Output:
start: drop = 0.000 loss = 185.81
step 4: drop = 1.121 loss = 35.98
step 8: drop = 1.611 loss = 7.28
step 12: drop = 1.825 loss = 1.78
step 16: drop = 1.919 loss = 0.72
step 20: drop = 1.960 loss = 0.52
The loss falls step by step and drop climbs towards 2 cups per degree, with no brute force. Why subtract? If the loss goes up as drop grows, the slope is positive, so we move drop down; if the slope is negative, we move it up. Always against the slope.
The learning rate: how big is each step?
| On the hill | Learning rate | Result |
|---|---|---|
| Tiny steps | too small | safe, but painfully slow |
| Sensible steps | just right | reaches the valley quickly |
| Giant leaps | too big | jumps over the valley, lands higher, and blows up |
Demo 3: too small, too big
days = [(14, 96), (18, 88), (22, 81), (26, 72), (30, 63), (34, 57)]
base = 76.17
def loss(drop):
return sum((base - drop * (t - 24) - cups) ** 2
for t, cups in days) / len(days)
def train(learning_rate, steps=20):
drop = 0.0
for _ in range(steps):
slope = (loss(drop + 0.001) - loss(drop)) / 0.001
drop = drop - learning_rate * slope
return drop
for lr in [0.0001, 0.002, 0.05]:
drop = train(lr)
print(f"learning rate {lr}: drop = {drop:.3g} loss = {loss(drop):.3g}")
Output:
learning rate 0.0001: drop = 0.341 loss = 128
learning rate 0.002: drop = 1.96 loss = 0.519
learning rate 0.05: drop = -3.89e+11 loss = 7.06e+24
- Too small: after 20 steps the loss is still 128. Crawling.
- Just right: loss about 0.5.
- Too big: no crash, no error, the loss is 7 × 10²⁴. Each step overshoots the bottom, the slope on the other side is steeper, and the next step is even bigger.
The learning rate is a setting we choose, not something the model learns. Such settings are called hyperparameters (episode 11 covers them in depth).
Demo 4: training both knobs, epochs and the loss curve
days = [(14, 96), (18, 88), (22, 81), (26, 72), (30, 63), (34, 57)]
def loss(base, drop): # now both knobs are learned
return sum((base - drop * (t - 24) - cups) ** 2
for t, cups in days) / len(days)
base, drop = 0.0, 0.0 # start anywhere
lr, h = 0.02, 0.001
print(f"start loss {loss(base, drop):7.1f}")
for epoch in range(1, 201): # epoch = one pass over all the days
slope_base = (loss(base + h, drop) - loss(base, drop)) / h
slope_drop = (loss(base, drop + h) - loss(base, drop)) / h
base -= lr * slope_base # both knobs step downhill together
drop -= lr * slope_drop
if epoch % 25 == 0:
bar = "#" * round(loss(base, drop) / 150)
print(f"epoch {epoch:3} loss {loss(base, drop):7.1f} {bar}")
print(f"learned: base {base:.1f}, drop {drop:.2f}")
print("20 °C ->", round(base - drop * (20 - 24)), "cups")
Output:
start loss 5987.2
epoch 25 loss 754.1 #####
epoch 50 loss 98.3 #
epoch 75 loss 13.2
epoch 100 loss 2.1
epoch 125 loss 0.7
epoch 150 loss 0.5
epoch 175 loss 0.5
epoch 200 loss 0.5
learned: base 76.1, drop 1.99
20 °C -> 84 cups
- One epoch = one pass over all the training data.
- The falling shape of the loss is the loss curve. When it flattens, training has converged.
- 20 °C → 84 cups: the same answer as episode 4, without trying every line. This exact loop scales to millions of knobs.
loss
6000 |#
|
750 | #
100 | #
13 | #
0.5 | # # # # ← flat: converged
+--------------------------- epochs
Three real-world details
| Term | What it means |
|---|---|
| Local minimum | A small dip in a bumpy loss landscape that isn't the lowest valley. With huge models this usually matters less than people fear, and some randomness helps escape. |
| Batch / stochastic gradient descent (SGD) | With millions of examples, each step uses a small random group (a batch) instead of the whole dataset. One epoch is then many steps. |
| Backpropagation | Real libraries compute all the slopes at once with calculus instead of nudging each knob. In neural networks this is called backpropagation (coming in the neural-network episodes). |
Glossary
| Term | Meaning |
|---|---|
| Parameter | A number the model learns (a knob), e.g. base, drop |
| Loss | One number measuring how wrong the model is |
| Mean squared error (MSE) | The average of the squared gaps between guess and truth |
| Loss landscape | The "hill": loss for every possible setting of the knobs |
| Slope / gradient | Which way, and how steeply, the loss changes when a knob moves |
| Gradient descent | Repeatedly stepping each knob against its slope to lower the loss |
| Learning rate | How big each step is (a hyperparameter) |
| Hyperparameter | A setting we choose before training, not learned (learning rate, number of epochs) |
| Step | One update of the knobs |
| Epoch | One full pass over the training data |
| Loss curve | The loss plotted over training steps or epochs |
| Converged | Training has settled: the loss stops going down |
| Local minimum | A dip that is lower than its surroundings but not the lowest point |
| Batch | A small group of examples used for one step |
| Stochastic gradient descent (SGD) | Gradient descent where each step uses a random batch |
| Backpropagation | Computing all the gradients of a neural network at once with calculus |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| Brute force on many knobs | 41 ** 1000 / 1_000_000 |
OverflowError: integer division result too large for a float |
| Learning rate too big | learning_rate = 0.05 |
no error, loss 7.06e+24 |
| Adding the slope instead of subtracting | drop = drop + learning_rate * slope |
no error, loss keeps rising |
| Learning rate too small | learning_rate = 0.0001 |
no error, loss still 128 after 20 steps |
Fixes: use gradient descent; if the loss grows, try a learning rate 10× smaller; always step against the slope; if the loss falls too slowly, raise the learning rate or train for more epochs.
Practice
- Find the largest learning rate that still works for Demo 3 (20 steps).
- Start
dropat10.0instead of0.0(learning rate 0.002). Does it still head for 2? - Swap
-for+in the step on purpose. What happens? - Print the loss every epoch in Demo 4 and sketch the loss curve.
Solutions
1. Try several values (the train function also takes a starting drop and a direction sign, used in questions 2 and 3):
days = [(14, 96), (18, 88), (22, 81), (26, 72), (30, 63), (34, 57)]
base = 76.17
def loss(drop):
return sum((base - drop * (t - 24) - cups) ** 2
for t, cups in days) / len(days)
def train(learning_rate, steps=20, drop=0.0, sign=-1):
for _ in range(steps):
slope = (loss(drop + 0.001) - loss(drop)) / 0.001
drop = drop + sign * learning_rate * slope
return drop
for lr in [0.01, 0.02, 0.021, 0.022, 0.03]:
d = train(lr)
print(f"learning rate {lr}: drop = {d:.3g} loss = {loss(d):.3g}")
d = train(0.002, drop=10.0)
print(f"start at 10: drop = {d:.3f} loss = {loss(d):.2f}")
d = train(0.002, sign=+1)
print(f"plus instead of minus: drop = {d:.3g} loss = {loss(d):.3g}")
learning rate 0.01: drop = 1.99 loss = 0.47
learning rate 0.02: drop = 1.88 loss = 1.08
learning rate 0.021: drop = 1.11 loss = 36.7
learning rate 0.022: drop = -3.64 loss = 1.48e+03
learning rate 0.03: drop = -2.54e+05 loss = 3.01e+12
start at 10: drop = 2.121 loss = 1.23
plus instead of minus: drop = -59.1 loss = 1.74e+05
0.01 is the best here; around 0.02 it still works but zig-zags; from about 0.022 it blows up. (For this loss the limit is 2 ÷ the bowl's curvature ≈ 0.0214.)
2. Yes: from 10 it walks down to 2.121 in 20 steps (loss 1.23) and would reach 2 with more steps. Gradient descent can start anywhere on a smooth bowl.
3. It walks uphill: drop goes to -59.1 and the loss to 174,000.
4. Change if epoch % 25 == 0: to print every epoch; the loss drops fast at first, then flattens near 0.5.
Quick quiz 1
The loss printed every epoch is 120, 340, 1900, 25000, 410000 ... What should you change first?
- A. Train for more epochs
- B. Lower the learning rate
- C. Add more knobs
- D. Use the test set for training
Answer
B, lower the learning rate. A loss that grows faster and faster means the steps overshoot the valley. More epochs would only make it worse.
Quick quiz 2
At drop = 1.0 the slope of the loss is -40, and learning_rate = 0.01. What is the new drop?
- A. 0.6
- B. 1.0
- C. 1.4
- D. -39.0
Answer
C, 1.4. new = old − learning_rate × slope = 1.0 − 0.01 × (−40) = 1.0 + 0.4. A negative slope means downhill is towards a bigger drop.
Interview questions
- What is gradient descent? Explain it without maths.
- Why can't we train a large model by trying every combination of parameters?
- What happens if the learning rate is too small? Too large? How do you spot each one?
- What is the difference between a step and an epoch?
- Why is mean squared error popular as a loss for gradient descent?
- What is a local minimum? What is stochastic (mini-batch) gradient descent and why is it used?
Summary
- A model is a box of knobs: its parameters. Training lowers the loss by turning them.
- Trying every setting takes centuries for 10 knobs and overflows for 1,000.
- Gradient descent: feel the slope of the loss, step against it, repeat.
- MSE squares the gaps, giving a smooth bowl to walk down.
- The learning rate sets the step size: too small crawls, too big explodes.
- Watch the loss curve over epochs; when it flattens, training has converged.
Next episode: our model is still a straight line. What if the pattern isn't a line at all? The artificial neuron: weights, bias and activation.