SyntaxPilot
Courses/Machine Learning/Lesson 8

Machine Learning · Lesson 8 of 90 · 11 min

Maths 6: gradient descent

loss function (cost function)objectiveoptimisationgradient descentlearning rateparameters vs hyperparametersconvergencedivergencelocal vs global minimumsaddle pointgradient descent from scratch

Machine Learning Full Course · Episode 8. Every code sample below was run with Python 3.13 in plain Python (no libraries); the output shown is the real output. The data is made up for teaching. You need episode 6 (derivatives) and episode 7 (partial derivatives and the gradient).

The problem: how much milk to buy?

Every morning at 5 am Raju buys milk for the day. Too little and customers walk away; too much and it spoils. He has noticed that more people walking past means more cups, and wrote down five days:

People walking past (hundreds) 1 2 3 4 5
Cups sold 32 55 79 98 124

He guesses a rule, a model: cups = w × people + b.

  • w: extra cups per hundred people (a weight)
  • b: a fixed base number of cups (a bias)

w and b are the model's parameters: numbers the model learns from data. Which w and b fit best?

Guessing by hand

people = [1, 2, 3, 4, 5]          # people walking past, in hundreds
cups = [32, 55, 79, 98, 124]      # cups Raju sold that day

def predict(w, b, x):
    return w * x + b

for w, b in [(20, 0), (25, 5), (22, 10)]:
    print(w, b, [predict(w, b, x) for x in people])

Output:

20 0 [20, 40, 60, 80, 100]
25 5 [30, 55, 80, 105, 130]
22 10 [32, 54, 76, 98, 120]

(20, 0) is too low everywhere, (25, 5) overshoots on busy days, (22, 10) is close but 4 short on the last day. Comparing five numbers by eye can't tell us which guess is best, and guessing never ends. We need one number for "how wrong".

The obvious "how wrong" fails

For each day, the miss (error) is prediction − actual. The obvious score: average the misses. Test it on a useless model: a flat line that always predicts 77.6 cups (the average), whatever the crowd.

people = [1, 2, 3, 4, 5]
cups = [32, 55, 79, 98, 124]

def avg_error(w, b):              # plain average of the misses
    errs = [w * x + b - c for x, c in zip(people, cups)]
    return sum(errs) / len(errs)

def loss(w, b):                   # average of the SQUARED misses
    errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
    return sum(errs) / len(errs)

print("flat line 77.6:", avg_error(0, 77.6), loss(0, 77.6))
print("good line:     ", avg_error(22, 10), loss(22, 10))

Output:

flat line 77.6: -5.684341886080802e-15 1032.24
good line:      -1.6 5.2
  • The flat line's plain average miss is −5.68 × 10⁻¹⁵: floating-point rounding noise, i.e. zero. A useless model scores "perfect".
  • Why: it predicts too many cups on quiet days (positive misses) and too few on busy days (negative misses), and they cancel.
  • The fix: square each miss first. A square is never negative, so nothing cancels, and big misses count much more. Flat line 1032.24, good line 5.2: now the score tells the truth.
  • zip(people, cups) walks through both lists in step: (1, 32), (2, 55), …

Loss function, objective, optimisation

Term Meaning Here
Loss function (also cost function) turns all of a model's mistakes into one number average of the squared misses
Objective what we want, stated precisely make the loss as small as possible
Optimisation searching for the parameters that achieve the objective find w, b with the lowest loss

Real-life example: a bowler's economy rate sums up a whole spell in one number. The average of squared errors is called mean squared error (MSE); episode 18 uses it by that name. Training a model is optimisation. Picture loss(w, b) as a landscape (w east, b north, loss = height): we want its lowest point.

First attempt: step by the full gradient (it explodes)

The gradient points uphill (episode 7), so step against it: w ← w − ∂loss/∂w, b ← b − ∂loss/∂b.

The partial derivatives of loss = (1/n) Σ (wx + b − c)² are, by the chain rule:

  • ∂loss/∂w = (1/n) Σ 2(wx + b − c)·x
  • ∂loss/∂b = (1/n) Σ 2(wx + b − c)
people = [1, 2, 3, 4, 5]
cups = [32, 55, 79, 98, 124]

def loss(w, b):
    errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
    return sum(errs) / len(errs)

def grad(w, b):                   # partial derivatives of the loss
    n = len(people)
    gw = sum(2 * (w * x + b - c) * x for x, c in zip(people, cups)) / n
    gb = sum(2 * (w * x + b - c) for x, c in zip(people, cups)) / n
    return gw, gb

w, b = 0.0, 0.0
for step in range(1, 301):
    gw, gb = grad(w, b)
    w, b = w - gw, b - gb         # step downhill by the full gradient
    if step <= 3 or step % 100 == 0:
        print(step, f"w={w:.3g} b={b:.3g} loss={loss(w, b):.3g}")

Output:

1 w=556 b=155 loss=3.62e+06
2 w=-1.21e+04 b=-3.34e+03 loss=1.86e+09
3 w=2.74e+05 b=7.58e+04 loss=9.55e+11
100 w=-7.97e+136 b=-2.21e+136 loss=8.08e+274
Traceback (most recent call last):
  File "main.py", line 19, in <module>
    print(step, f"w={w:.3g} b={b:.3g} loss={loss(w, b):.3g}")
                                            ~~~~^^^^^^
  File "main.py", line 5, in loss
    errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
            ~~~~~~~~~~~~~~~~^^~~
OverflowError: (34, 'Numerical result out of range')
  • f"{w:.3g}" prints 3 significant digits, switching to e-notation for huge numbers.
  • Why it fails: at (0, 0) the gradient is in the hundreds. The direction is right, but jumping hundreds of units flies over the valley and lands higher on the other side; each jump is bigger. Running away like this is called divergence. Python's floats can't go above about 1.8 × 10³⁰⁸, so squaring 10¹³⁷ raises OverflowError.

The learning rate

Real-life example: walking down a hill in thick fog. You can't see the valley, only feel the slope under your feet (the gradient), so you choose your stride. That stride is the learning rate η (Greek "eta"): a small number the gradient is multiplied by.

Gradient descent: repeat w ← w − η · ∂loss/∂w (and the same for b) until the ground is flat.

people = [1, 2, 3, 4, 5]
cups = [32, 55, 79, 98, 124]

def loss(w, b):
    errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
    return sum(errs) / len(errs)

def grad(w, b):
    n = len(people)
    gw = sum(2 * (w * x + b - c) * x for x, c in zip(people, cups)) / n
    gb = sum(2 * (w * x + b - c) for x, c in zip(people, cups)) / n
    return gw, gb

for lr in [0.001, 0.05, 0.1]:     # the learning rate
    w, b = 0.0, 0.0
    for step in range(200):
        gw, gb = grad(w, b)
        w, b = w - lr * gw, b - lr * gb
    print(f"lr={lr}: w={w:.4g} b={b:.4g} loss={loss(w, b):.4g}")

Output:

lr=0.001: w=23.28 b=6.657 loss=3.563
lr=0.05: w=22.73 b=9.401 loss=1.662
lr=0.1: w=-2.977e+28 b=-8.246e+27 loss=1.129e+58
Learning rate What happened Name
0.001 still walking down after 200 steps too small: slow
0.05 settled at the bottom convergence
0.1 blew up divergence

Converged means the steps no longer change anything (the gradient is ≈ 0). Doubling 0.05 to 0.1 flips the result from convergence to divergence: the learning rate is the most important setting in gradient descent.

Why there is a limit: on a bowl-shaped loss, a step overshoots the bottom when η is larger than about 2 ÷ (the steepest curvature). For this data that limit is ≈ 0.0845 (see practice 1).

Parameters vs hyperparameters

Parameters Hyperparameters
Examples w, b (later: millions of neural-network weights) learning rate, number of steps, tolerance
Who sets them learned from the data chosen by you before training
Analogy how the cake turns out the oven temperature you set

Later in the course (episode 42) we tune hyperparameters automatically.

Convergence: a stopping rule

people, cups = [1, 2, 3, 4, 5], [32, 55, 79, 98, 124]
def grad(w, b):
    n = len(people)
    gw = sum(2 * (w * x + b - c) * x for x, c in zip(people, cups)) / n
    gb = sum(2 * (w * x + b - c) for x, c in zip(people, cups)) / n
    return gw, gb

def gradient_descent(lr=0.05, tol=1e-6, max_steps=10_000):
    w, b = 0.0, 0.0                       # parameters: learned
    for step in range(1, max_steps + 1):  # lr, tol, max_steps: hyperparameters
        gw, gb = grad(w, b)
        w, b = w - lr * gw, b - lr * gb
        if (gw ** 2 + gb ** 2) ** 0.5 < tol:   # nearly flat: converged
            return w, b, step
    return w, b, max_steps
w, b, steps = gradient_descent()
print(f"w={w:.4f} b={b:.4f} after {steps} steps")
print("100 people ->", round(w * 1 + b), "cups; 600 ->", round(w * 6 + b))

Output:

w=22.7000 b=9.5000 after 815 steps
100 people -> 32 cups; 600 -> 146
  • Tolerance (tol): how flat counts as flat. We stop when the gradient's length √(gw² + gb²) < 10⁻⁶.
  • max_steps is a safety net in case it never converges.
  • 10_000 is just 10000; Python allows underscores in numbers for readability.
  • The answer, w = 22.7 and b = 9.5, is the exact best line (the same as the formula used in episode 18).

Local vs global minimum

Gradient descent only feels the ground under its feet. On the hilly function from episode 6, f(x) = x⁴ − 4x² + x, the starting point decides which valley you end in:

def hills(x):
    return x ** 4 - 4 * x ** 2 + x

def d_hills(x):
    return 4 * x ** 3 - 8 * x + 1

for start in [1.0, -2.0]:
    x = start
    for _ in range(500):
        x = x - 0.01 * d_hills(x)
    print(f"start {start:+}: ended at x={x:.3f}, height {hills(x):.2f}")

Output:

start +1.0: ended at x=1.347, height -2.62
start -2.0: ended at x=-1.473, height -5.44
  • Local minimum: lowest only in its neighbourhood (x ≈ 1.347, −2.62).
  • Global minimum: lowest of all (x ≈ −1.473, −5.44).
  • f"{start:+}" always prints the sign.

Raju's loss (and linear and logistic regression losses) is convex, a single bowl, so any start reaches the global minimum. Neural-network losses are non-convex; common remedies are several random starts and noisy steps (stochastic gradient descent, episode 18).

Saddle points

A horse's saddle curves up from front to back and down from side to side. f(x, y) = x² − y² has this shape: at (0, 0) the gradient is zero, yet it is not a minimum. That is a saddle point.

def saddle(x, y):
    return x ** 2 - y ** 2

def grad(x, y):
    return 2 * x, -2 * y

for x, y in [(1.0, 0.0), (1.0, 0.001)]:
    for step in range(1, 1001):
        gx, gy = grad(x, y)
        x, y = x - 0.01 * gx, y - 0.01 * gy
        if step in (200, 600, 1000):
            print(step, f"x={x:.4f} y={y:.4f} f={saddle(x, y):.4f}")
    print("---")

Output:

200 x=0.0176 y=0.0000 f=0.0003
600 x=0.0000 y=0.0000 f=0.0000
1000 x=0.0000 y=0.0000 f=0.0000
---
200 x=0.0176 y=0.0525 f=-0.0024
600 x=0.0000 y=144.5783 f=-20902.8794
1000 x=0.0000 y=398264.6517 f=-158614732760.3709
---

Starting exactly on y = 0, it slides to (0, 0) and stops for good (every step is zero). A nudge of 0.001 in y grows and escapes, and f keeps falling (this function has no minimum at all). A zero gradient can be a minimum, a maximum or a saddle point. In high dimensions saddle points are very common; noise in the steps helps escape them.

Common mistakes

Mistake Result Why Fix
average the plain misses -5.684341886080802e-15 for a useless line positive and negative misses cancel square them (the loss)
w, b = w - gw, b - gb OverflowError: (34, 'Numerical result out of range') full steps overshoot further each time: divergence multiply by a learning rate
lr = 0.1 on this data loss 1.129e+58 above the stability limit (≈ 0.0845) lower it; watch the loss each step
lr = 0.001 loss still 3.563 after 200 steps steps too small raise it; try 0.01, 0.03, 0.05
no stopping rule wasted steps or an endless loop nothing checks convergence stop when the gradient's length < tol, plus max_steps
gradient ≈ 0 ⇒ best stuck at −2.62, or on a saddle local minimum / saddle point several starts; check the result
loss = sum(errs) without dividing by n the gradient grows with the data size, so a good lr stops working the loss depends on n average (divide by n)

Mini project: Raju's best price and sugar, by gradient descent

Episode 7's profit(p, s) has two knobs. To maximise profit, minimise minus profit with a reusable gradient-descent function:

def gradient_descent(grad, start, lr, steps):
    x = list(start)
    for _ in range(steps):
        g = grad(*x)
        x = [xi - lr * gi for xi, gi in zip(x, g)]
    return x

def neg_profit_grad(p, s):                 # gradient of MINUS profit
    m = p - 5 - s
    c = 200 - 5 * p + 20 * s - 5 * s ** 2
    return -(c - 5 * m), -(-c + m * (20 - 10 * s))

p, s = gradient_descent(neg_profit_grad, [20, 1], lr=0.005, steps=2000)
c = 200 - 5 * p + 20 * s - 5 * s ** 2
print(f"best price Rs {p:.2f}, sugar {s:.2f} spoons, profit Rs {(p - 5 - s) * c:.2f}")
print("gradient there:", [round(g, 6) for g in neg_profit_grad(p, s)])

Output:

best price Rs 25.12, sugar 1.50 spoons, profit Rs 1734.45
gradient there: [-0.0, 0.0]

The gradient is 0 at the answer (-0.0 is just zero with a minus sign from rounding): Rs 25.12 a cup with one and a half spoons, up from Rs 1,610 at (20, 1). Try lr = 0.05: it diverges.

Practice

  1. Find the largest learning rate that still converges on Raju's data.
Solution

Run 1000 steps for several values:

people = [1, 2, 3, 4, 5]
cups = [32, 55, 79, 98, 124]

def loss(w, b):
    errs = [(w * x + b - c) ** 2 for x, c in zip(people, cups)]
    return sum(errs) / len(errs)

def grad(w, b):
    n = len(people)
    gw = sum(2 * (w * x + b - c) * x for x, c in zip(people, cups)) / n
    gb = sum(2 * (w * x + b - c) for x, c in zip(people, cups)) / n
    return gw, gb

for lr in [0.08, 0.084, 0.085, 0.09]:
    w, b = 0.0, 0.0
    for step in range(1000):
        gw, gb = grad(w, b)
        w, b = w - lr * gw, b - lr * gb
    print(f"lr={lr}: loss={loss(w, b):.4g}")

Output:

lr=0.08: loss=1.66
lr=0.084: loss=1.66
lr=0.085: loss=3.762e+13
lr=0.09: loss=4.738e+109

The limit is between 0.084 and 0.085. (Theory: the loss's Hessian is [[22, 6], [6, 2]], its largest eigenvalue ≈ 23.66, and 2 ÷ 23.66 ≈ 0.0845.)

  1. Add a sixth day: 7 hundred people, 168 cups. Retrain.
Solution

With people = [1, 2, 3, 4, 5, 7], cups = [32, 55, 79, 98, 124, 168] and 2000 steps, both lr = 0.05 and lr = 0.03 give w=22.64 b=9.643 loss=1.393. Bigger inputs make the loss steeper, so the stability limit drops (to about 0.055 here): 0.05 is now close to the edge.

  1. Gradient descent on loss(w) = (w − 3)² + 2 with lr = 0.1 from w = 0.
Solution
w = 0.0
for step in range(1, 51):
    w = w - 0.1 * 2 * (w - 3)          # loss = (w - 3) ** 2 + 2
    if step in (1, 2, 10, 50):
        print(step, round(w, 4), round((w - 3) ** 2 + 2, 4))

Output:

1 0.6 7.76
2 1.08 5.6864
10 2.6779 2.1038
50 3.0 2.0

It converges to w = 3, loss 2. The "+ 2" shifts the height, not the location of the minimum.

  1. Run the hills from 10 random starts between −2 and 2.
Solution
import random

def d_hills(x):
    return 4 * x ** 3 - 8 * x + 1

random.seed(3)
ends = []
for _ in range(10):
    x = random.uniform(-2, 2)
    for _ in range(500):
        x = x - 0.01 * d_hills(x)
    ends.append(round(x, 3))
print(ends)
print("global:", sum(e < 0 for e in ends), "local:", sum(e > 0 for e in ends))

Output:

[-1.473, 1.347, -1.473, 1.347, 1.347, -1.473, -1.473, 1.347, -1.473, -1.473]
global: 6 local: 4

Keep the best of several starts: this is the "random restarts" trick. random.seed(3) makes the "random" starts repeatable.

Quick quiz

def grad(w):
    return 2 * (w - 3)      # loss = (w - 3) ** 2

w = 0
for _ in range(2):
    w = w - 0.5 * grad(w)
print(w)

A) 3.0 B) 1.5 C) 6.0 D) 0

Answer

A) 3.0. Step 1: grad(0) = −6, so w = 0 + 3 = 3.0, exactly the bottom. Step 2: grad(3.0) = 0, so w stays 3.0. With lr = 1, w would jump 0 → 6 → 0 → 6 forever.

Interview questions

  1. What is gradient descent? An iterative optimisation algorithm: repeatedly move the parameters a small step (learning rate × gradient) against the gradient of the loss until it converges.
  2. What happens if the learning rate is too high or too low? Too high: overshooting, oscillation or divergence (NaN/overflow). Too low: very slow convergence, possibly stopping early in a poor spot.
  3. Loss function vs cost function vs objective? Often used interchangeably; strictly, a loss is per example, the cost is the average over the dataset (plus any penalties), and the objective is whatever is optimised (minimised or maximised).
  4. Why squared error rather than plain error? Plain errors cancel; squares are always positive, punish big errors more, and are smooth (easy to differentiate).
  5. Parameters vs hyperparameters? Parameters are learned from data (weights, biases); hyperparameters are set before training (learning rate, number of epochs, regularisation strength).
  6. How do you know gradient descent has converged? The gradient norm or the change in loss falls below a tolerance, or the validation loss stops improving.
  7. Local minimum vs saddle point? Both have zero gradient; a local minimum curves up in every direction, a saddle curves up in some and down in others. In high dimensions saddle points are far more common.
  8. Does gradient descent always find the global minimum? Only for convex losses (with a suitable learning rate); otherwise it can stop in a local minimum or slow down near a saddle.

Glossary

Term Meaning Example
Model A rule that turns inputs into a prediction cups = w·x + b
Weight (w) A learned multiplier for an input 22.7 cups per hundred people
Bias (b) A learned constant added at the end 9.5 cups
Parameter A number the model learns from data w, b
Error (miss) prediction − actual 120 − 124 = −4
Loss function / cost function One number measuring how wrong the model is average squared miss
Mean squared error (MSE) The average of squared errors 5.2 for (22, 10)
Objective What we want, stated precisely minimise the loss
Optimisation Searching for the parameters that achieve the objective gradient descent
Gradient descent Repeatedly stepping against the gradient w ← w − η·∂L/∂w
Learning rate (η, lr) The step-size multiplier 0.05
Divergence Steps growing until the numbers blow up lr = 0.1 → 1e+58
Convergence Steps settling at a point where nothing changes w = 22.7, b = 9.5
Tolerance (tol) How small the gradient must get to count as flat 1e-6
Hyperparameter A setting chosen before training lr, tol, max_steps
OverflowError Python error: a number too big for a float (34, 'Numerical result out of range')
Floating-point noise Tiny rounding leftovers −5.68e-15 ≈ 0
Local minimum Lowest only in its neighbourhood x ≈ 1.347
Global minimum Lowest point of all x ≈ −1.473
Convex / non-convex One bowl / many valleys line fit / neural network
Saddle point Zero gradient, curving up one way and down another (0, 0) of x² − y²
Random restarts Running from several starts and keeping the best practice 4
random.seed(n) Makes Python's random numbers repeatable random.seed(3)
zip(a, b) Walks through two lists in step zip(people, cups)

Summary

  • A loss function turns all the mistakes into one number; squaring stops misses from cancelling.
  • Training = optimisation: find the parameters with the lowest loss.
  • Gradient descent steps against the gradient: w ← w − η·gradient, repeated until it converges.
  • The learning rate decides everything: too small crawls, too big diverges (real OverflowError).
  • Parameters are learned; hyperparameters (lr, steps, tolerance) are chosen.
  • Watch out for local minima and saddle points: a zero gradient isn't always the best.

Next episode: Maths for ML 7, probability basics: sample spaces, events, conditional probability, independence and Bayes' theorem.