Machine Learning Full Course · Episode 6. Every code sample below was run with Python 3.13 in plain Python (no libraries); the output shown is the real output. No calculus background is assumed: only arithmetic and powers like x² and x³.
The problem: the best price for a cup of chai
Each cup costs Raju Rs 5 to make. At Rs 10 a cup, 150 people buy but he keeps only Rs 5 each; at Rs 40 nobody buys. Every extra rupee loses 5 customers. Somewhere in between is the price with the most profit. Machine learning has the same problem upside down: find the settings that give the least error.
Functions
A function is a rule that takes an input and always gives exactly one output, like a sugarcane juice machine (cane in, juice out). Written f(x), "f of x".
- cups(p) = 200 − 5p
- profit(p) = (p − 5) × cups(p)
def cups(p): # cups sold at price p
return 200 - 5 * p
def profit(p): # Rs 5 cost per cup
return (p - 5) * cups(p)
for p in [10, 20, 30, 40]:
print(p, cups(p), profit(p))
Output:
10 150 750
20 100 1500
30 50 1250
40 0 0
Profit rises, then falls: the best price is somewhere between 20 and 30.
The obvious way: try every value (and why it fails)
import time
def profit(p):
return (p - 5) * (200 - 5 * p)
best = max(range(5, 41), key=profit)
print("best price:", best, profit(best))
t = time.perf_counter()
for _ in range(1_000_000):
profit(22)
per_try = (time.perf_counter() - t) / 1_000_000
tries = 100 ** 10 # 10 knobs, 100 values each
years = tries * per_try / (3600 * 24 * 365)
print(f"{tries:.0e} tries = {years:.0e} years")
Output (one run; the second number depends on your computer):
best price: 22 1530
1e+20 tries = 4e+05 years
max(range(5, 41), key=profit)tries every whole price and keeps the one with the biggest profit: Rs 22.- A model doesn't have one knob (a number we can tune) but thousands or millions. Each new knob multiplies the combinations by 100, so 10 knobs already need 10²⁰ tries: hundreds of thousands of years. This blow-up is called exponential growth.
Why it fails: blind search never uses the shape of the function. We need to stand at one point and know which way to move.
Slope and rate of change
| Idea | Real life | Maths |
|---|---|---|
| Slope | a ghat road sign "1 in 10": 1 m up per 10 m along | rise / run = 1/10 = 0.1 |
| Rate of change | a speedometer: 60 km per hour, right now | change in output per unit change of input |
| Sign | walking uphill, downhill, on flat ground | + up · − down · 0 flat |
For Raju, a positive slope of profit means raising the price raises the profit.
The derivative
A curve's steepness changes from place to place, so measure the slope over a small step h: (f(x + h) − f(x)) / h, and shrink h.
def profit(p):
return (p - 5) * (200 - 5 * p)
def slope(f, x, h): # rise over run, over a small step h
return (f(x + h) - f(x)) / h
for h in [1, 0.1, 0.001]:
print(h, slope(profit, 10, h))
print("tiny h:", slope(profit, 10, 1e-15))
Output:
1 120.0
0.1 124.49999999999932
0.001 124.99499999989894
tiny h: 227.37367544323203
The values close in on 125. The value the slope approaches as h shrinks towards 0 (its limit) is the derivative: the slope at exactly one point. At Rs 10, profit′ = 125: each extra rupee adds about Rs 125 of profit.
The tiny-h trap: h = 10⁻¹⁵ gives 227, nearly double. Computers store decimals with about 16 significant digits (floating point). f(x + h) − f(x) subtracts two almost equal numbers, leaving only rounding noise, and dividing by a tiny h magnifies it. Too big an h is inaccurate; too small is noise.
Numerical derivative: the central difference
def profit(p):
return (p - 5) * (200 - 5 * p)
def slope(f, x, h=1e-5): # central difference
return (f(x + h) - f(x - h)) / (2 * h)
print(slope(profit, 10))
Output:
125.00000000272847
Stepping to both sides and dividing by 2h (the central difference) is far more accurate; h ≈ 10⁻⁵ is a good default. A derivative computed this way is a numerical derivative. Use it to check every derivative you work out by hand (ML libraries call this gradient checking).
Derivative rules
Notation: f′(x) ("f prime of x") or d/dx f(x) both mean the derivative of f.
| Rule | Formula | Example |
|---|---|---|
| Power rule | (xⁿ)′ = n·xⁿ⁻¹ | (x⁴)′ = 4x³; (x²)′ = 2x; (x)′ = 1 |
| Constant | (c)′ = 0 | (7)′ = 0 |
| Constant multiple | (c·f)′ = c·f′ | (7x)′ = 7 |
| Sum rule | (f + g)′ = f′ + g′ | (x² + 7x)′ = 2x + 7 |
| Product rule | (f·g)′ = f′·g + f·g′ | (x²(x + 1))′ = 2x(x + 1) + x² |
def slope(f, x, h=1e-5):
return (f(x + h) - f(x - h)) / (2 * h)
def power(t): return t ** 4
def total(t): return t ** 2 + 7 * t
def prod(t): return t ** 2 * (t + 1)
x = 3
print("power: ", round(slope(power, x), 3), 4 * x ** 3)
print("sum: ", round(slope(total, x), 3), 2 * x + 7)
print("product:", round(slope(prod, x), 3), 2 * x * (x + 1) + x ** 2 * 1)
Output:
power: 108.0 108
sum: 13.0 13
product: 33.0 33
Raju's profit by the product rule: f = p − 5 (f′ = 1), g = 200 − 5p (g′ = −5): profit′ = 1·(200 − 5p) + (p − 5)·(−5) = 200 − 5p − 5p + 25 = −10p + 225. At p = 10: 125, matching the measurement.
The chain rule
A function inside a function: f(x) = (3x + 1)². Think of cycle gears: the pedal gear turns the middle gear 3× as fast (inner function u = 3x + 1, rate du/dx = 3), and the middle gear drives the wheel (outer function u², rate 2u). Pedal-to-wheel rate: multiply the rates.
Chain rule: (f(g(x)))′ = f′(g(x)) · g′(x), "outer rate × inner rate". In Leibniz notation: dy/dx = dy/du · du/dx.
def f(x):
return (3 * x + 1) ** 2
def slope(f, x, h=1e-5):
return (f(x + h) - f(x - h)) / (2 * h)
x = 2
print("power rule only:", 2 * (3 * x + 1))
print("chain rule:", 2 * (3 * x + 1) * 3)
print("numerical:", round(slope(f, x), 4))
Output:
power rule only: 14
chain rule: 42
numerical: 42.0
Forgetting the inner rate is the most common derivative mistake; here it is off by exactly the factor 3. Neural networks are hundreds of functions nested inside each other, and training them (backpropagation, later in the course) is the chain rule applied over and over.
Maximum, minimum and setting the derivative to zero
Walking over a hill: before the top the slope is +, exactly at the top it is 0, after it is −. The highest point is the maximum; the lowest point of a valley is the minimum. At both, the derivative is 0 (a stationary point or flat spot). To find them: set the derivative to zero and solve.
For Raju: −10p + 225 = 0 ⇒ p = 22.5.
def profit(p):
return (p - 5) * (200 - 5 * p)
def slope(f, x, h=1e-5):
return (f(x + h) - f(x - h)) / (2 * h)
def d_profit(p): # from the rules: -10p + 225
return -10 * p + 225
for p in [10, 22.5, 30]:
print(p, round(slope(profit, p), 4), d_profit(p))
print("best:", 225 / 10, profit(22.5))
Output:
10 125.0 125
22.5 0.0 0.0
30 -75.0 -75
best: 22.5 1531.25
+125 (go up), 0 (the top), −75 (too far). Best price Rs 22.50, profit Rs 1,531.25, slightly better than brute force's whole-rupee answer (Rs 22, 1,530), found with one equation instead of a loop.
How to tell a maximum from a minimum: the slope goes + → 0 → − at a maximum and − → 0 → + at a minimum (the second derivative, the slope of the slope, is negative at a maximum and positive at a minimum; more in episode 7).
Convex vs non-convex functions
| Convex | Non-convex | |
|---|---|---|
| Picture | a round bowl: a marble always rolls to the one bottom | a mountain range: the marble stops in the nearest valley |
| Formal test | a straight line between any two points of the curve never goes below the curve | some such line dips below |
| Flat spots | one, and it is the best (the global minimum) | several: local minima, maxima |
| In ML | linear and logistic regression losses | neural network losses |
A local minimum is the lowest point only in its neighbourhood; the global minimum is the lowest of all.
def bowl(x):
return (x - 3) ** 2 + 1
def hills(x):
return x ** 4 - 4 * x ** 2 + x
def slope(f, x, h=1e-5):
return (f(x + h) - f(x - h)) / (2 * h)
for name, f in [("bowl", bowl), ("hills", hills)]:
xs = [i / 100 + 0.005 for i in range(-300, 500)]
flat = [round(x, 1) for x in xs if slope(f, x) * slope(f, x + 0.01) < 0]
print(name, "slope = 0 near:", [(x, round(f(x), 2)) for x in flat])
Output:
bowl slope = 0 near: [(3.0, 1.0)]
hills slope = 0 near: [(-1.5, -5.44), (0.1, 0.06), (1.3, -2.6)]
The loop walks along x from −3 to 5 in steps of 0.01 and records where the slope changes sign (product of neighbouring slopes < 0). The bowl has one flat spot (its minimum, 1.0 at x = 3). The hills have three: the global minimum (−5.44 at x ≈ −1.5), a local maximum (x ≈ 0.1) and a local minimum (−2.6 at x ≈ 1.3) that is not the lowest. Rolling downhill from x = 1 gets stuck at 1.3.
Common mistakes
| Mistake | Result | Why | Fix |
|---|---|---|---|
| try every value of every knob | 1e+20 tries = 100,000+ years | combinations multiply with each knob | follow the derivative |
slope(profit, 10, 1e-15) |
227.37367544323203 instead of 125 |
floating-point rounding noise ÷ tiny h | central difference, h ≈ 1e-5 |
| ((3x + 1)²)′ = 2(3x + 1) | 14 instead of 42 at x = 2 | forgot the inner rate | chain rule: × 3 |
| (x²)′ = x | wrong slope everywhere | power rule: bring the power down AND subtract 1 | (x²)′ = 2x |
| (f·g)′ = f′·g′ | wrong for products | the product rule has two terms | f′g + fg′ |
| slope 0 ⇒ the best point | stuck at x = 1.3 (−2.6) | non-convex: local minimum or even a peak | check the shape; compare candidates |
Mini project: let the slope find the price
Instead of solving the equation, start anywhere and keep stepping a little in the uphill direction:
def profit(p):
return (p - 5) * (200 - 5 * p)
def slope(f, x, h=1e-5):
return (f(x + h) - f(x - h)) / (2 * h)
p = 10.0 # start anywhere
for step in range(1, 31):
p = p + 0.05 * slope(profit, p) # move uphill, a little
if step % 5 == 0:
print(step, round(p, 4), round(profit(p), 2))
Output:
5 22.1094 1530.49
10 22.4878 1531.25
15 22.4996 1531.25
20 22.5 1531.25
25 22.5 1531.25
30 22.5 1531.25
Big slope, big step; as the slope flattens near the top, the steps shrink. Flip the sign (step downhill) and minimise an error instead, and you have gradient descent, which we build properly in episode 8. Try a step size of 0.3 instead of 0.05: each step overshoots further (47.5, −27.5, 122.5, …) and it blows up; the step size matters (the learning rate, episode 8).
Practice
- Derivative of 4x³ − 2x + 9; check it numerically at x = 2.
Solution
Power, constant-multiple, sum and constant rules: 12x² − 2. At x = 2: 46. round(slope(lambda x: 4*x**3 - 2*x + 9, 2), 3) → 46.0. (lambda x: ... is Python's way to write a one-line function without def.)
- Raju's cost rises to Rs 7 a cup. New best price?
Solution
profit = (p − 7)(200 − 5p); profit′ = (200 − 5p) − 5(p − 7) = −10p + 235 = 0 ⇒ p = 23.5, profit = Rs 1,361.25. The numerical slope at 23.5 prints 0.0.
- Derivative of (2x − 5)³ by the chain rule; check at x = 4.
Solution
Outer u³ → 3u², inner 2x − 5 → 2: 3(2x − 5)²·2 = 6(2x − 5)². At x = 4: 6 × 9 = 54; numerically 54.0.
- Find the flat spots of x³ − 3x.
Solution
3x² − 3 = 0 ⇒ x = ±1. Numerical slope at both is 0.0. x = −1 is a local maximum (f = 2), x = 1 a local minimum (f = −2). The function isn't convex.
Quick quiz
def f(x):
return x ** 3 - 12 * x
def df(x):
return 3 * x ** 2 - 12
print(df(2), df(3))
A) 0 15 B) -4 9 C) 0 -3 D) 12 15
Answer
A) 0 15. df(2) = 12 − 12 = 0, a flat spot (here a local minimum); df(3) = 27 − 12 = 15, going up.
Interview questions
- What is a derivative, intuitively? The instantaneous rate of change: the slope of the function at one point.
- Why does ML need derivatives? Training minimises a loss; the derivative (gradient) says which direction lowers the loss and by how much, without blind search.
- State the chain rule and why it matters for neural networks. (f(g(x)))′ = f′(g(x))·g′(x); backpropagation applies it layer by layer to get every weight's gradient.
- Forward vs central difference? Forward (f(x+h) − f(x))/h has error proportional to h; central (f(x+h) − f(x−h))/2h has error proportional to h², so it is much more accurate for the same h.
- Why not use a tiny h like 1e-15? Floating-point cancellation: rounding error divided by h dominates.
- What is a convex function and why is it nice? Any chord lies above the curve; every local minimum is global, so gradient methods find the best answer.
- Does a zero derivative guarantee a minimum? No: it can be a maximum, a local minimum or a saddle/inflection point (like x³ at 0).
Glossary
| Term | Meaning | Example |
|---|---|---|
| Function | A rule giving exactly one output for each input | profit(p) |
| Input / output | What goes in / what comes out of a function | price → profit |
| Knob (parameter) | A number we can tune | the price; a model's weights |
| Brute force (blind search) | Trying every possible value | max(range(5, 41), key=profit) |
| Exponential growth | Multiplying by a fixed factor each step | 100¹⁰ = 10²⁰ |
| Slope | Rise ÷ run | "1 in 10" = 0.1 |
| Rate of change | Change in output per unit change of input | 60 km per hour |
| Limit | The value something approaches as a step shrinks to 0 | slope → 125 |
| Derivative (f′, d/dx) | The slope at one exact point | profit′(10) = 125 |
| Floating point | How computers store decimals, about 16 digits | tiny h gives 227 |
| Forward difference | (f(x + h) − f(x)) / h | first slope function |
| Central difference | (f(x + h) − f(x − h)) / 2h | accurate numerical slope |
| Numerical derivative | A derivative measured by computing | slope(profit, 10) |
| Constant | A plain number that never changes | 7 |
| Power rule | (xⁿ)′ = n·xⁿ⁻¹ | (x⁴)′ = 4x³ |
| Sum rule | (f + g)′ = f′ + g′ | (x² + 7x)′ = 2x + 7 |
| Constant multiple rule | (c·f)′ = c·f′ | (7x)′ = 7 |
| Product rule | (f·g)′ = f′g + fg′ | profit′ = −10p + 225 |
| Inner / outer function | In f(g(x)), g is inner and f is outer | 3x + 1 inside u² |
| Chain rule | Derivative of nested functions = outer rate × inner rate | ((3x + 1)²)′ = 6(3x + 1) |
| Backpropagation | Training a neural network by the chain rule | later in the course |
| Maximum / minimum | Highest / lowest point | Rs 22.50 is the maximum of profit |
| Stationary point | Where the derivative is 0 | x = 22.5 |
| Convex function | Bowl-shaped; any chord lies above the curve; one minimum | (x − 3)² + 1 |
| Non-convex function | Has several valleys or peaks | x⁴ − 4x² + x |
| Local minimum | Lowest only in its neighbourhood | x ≈ 1.3 |
| Global minimum | Lowest of all | x ≈ −1.5 |
| Gradient descent | Repeatedly stepping downhill using the slope | episode 8 |
lambda |
A one-line Python function without a name | lambda x: x ** 2 |
Summary
- A function maps each input to one output; ML wants the input (weights) with the smallest output (error).
- Brute force explodes with every extra knob; we need to know which way to move.
- The derivative is the slope at a point, the rate of change; measure it with the central difference (h ≈ 1e-5, never tiny).
- Power, sum, constant-multiple and product rules give derivatives on paper; the chain rule multiplies rates through nested functions.
- Maxima and minima have zero slope: set f′ = 0 and solve (Raju: Rs 22.50).
- Convex functions have one bottom; non-convex ones have local minima where a zero slope isn't the best.
Next episode: Maths for ML 5, partial derivatives and gradients: many knobs at once, the gradient vector that points uphill, and the chain rule on a computational graph.