Machine Learning Full Course · Episode 10. Every code sample below was run with Python 3.13, NumPy 2.5.3 and SciPy 1.18.1; the output shown is the real output. All the data (the 0.3 buy rate, 4 customers per 10 minutes, the heights) is made up for teaching.
The problem: how many cups tomorrow?
Raju buys milk every evening for the next day. Too little and he turns customers away; too much and it spoils. So he asks: how many cups will I sell tomorrow?
There is no single answer: some days 3 cups, some days 9. Tomorrow's sales are a random variable (Episode 9: a number that depends on chance). What Raju needs is the chance of every possible value: P(0 cups), P(1 cup), …, P(20 cups).
A list (or formula) giving the probability of every value of a random variable is a probability distribution. Today we invent the most important ones.
Discrete vs continuous random variables
| Kind | Values | Examples | Tool |
|---|---|---|---|
| Discrete | countable, with gaps | cups sold (0, 1, 2 … never 2.5), customers per hour | PMF: a probability for each value |
| Continuous | any value in a range, no gaps | height 165.37 cm, waiting time 2.81 min | PDF: a density; probability = area |
Bernoulli: one yes-or-no try
One person walks past: they buy (1, a success) or don't (0, a failure). A single try with two outcomes is a Bernoulli trial. If 3 in 10 people buy, then P(1) = p = 0.3 and P(0) = 1 − p = 0.7. The Bernoulli distribution is described by that one number p.
import numpy as np
rng = np.random.default_rng(10)
buys = rng.random(10) < 0.3 # 10 passers-by, each buys with chance 0.3
print(buys.astype(int))
print("cups sold:", buys.sum())
more = [int((rng.random(10) < 0.3).sum()) for _ in range(5)]
print("cups sold, 5 more days:", more)
Output:
[0 1 0 1 0 1 0 0 0 0]
cups sold: 3
cups sold, 5 more days: [2, 2, 6, 2, 1]
np.random.default_rng(10)creates a random number generator; the seed 10 makes the "random" numbers repeat exactly, so you get the same output.rng.random(10)gives 10 numbers between 0 and 1;< 0.3isTrue30% of the time, which is one Bernoulli trial per person.astype(int)turnsTrue/Falseinto1/0;sum()counts the buyers.
Same p, different days: 3, 2, 2, 6, 2, 1 cups. How likely is each count?
Binomial: the obvious way fails
20 people walk past. P(exactly 6 cups)? The obvious way is to list every buy/no-buy pattern and add up the chances of the patterns with 6 buys:
import itertools, time
p, n = 0.3, 20
start = time.time()
probs = [0.0] * (n + 1)
for day in itertools.product([0, 1], repeat=n): # every buy/no-buy pattern
k = sum(day)
probs[k] += p**k * (1 - p)**(n - k)
secs = time.time() - start
print("P(exactly 6 cups) =", round(probs[6], 4))
print("patterns:", 2**n, " seconds:", round(secs, 1))
years = 2**100 / (2**n / secs) / 31_557_600
print("100 people would take about", f"{years:.1e}", "years")
Output:
P(exactly 6 cups) = 0.1916
patterns: 1048576 seconds: 0.5
100 people would take about 2.1e+16 years
It works (0.1916), but needs 2²⁰ ≈ one million patterns for 20 people, and every extra person doubles the work. For 100 people it would take more than ten thousand trillion years (the exact estimate depends on your computer's speed). itertools.product([0, 1], repeat=n) produces every pattern of n zeros and ones.
Why it fails: we list patterns one by one, when all we need is how many there are.
Binomial: the fix (count the ways)
- Every pattern with exactly k buys has the same chance: pᵏ × (1 − p)ⁿ⁻ᵏ.
- The number of such patterns is "n choose k", the combinations of k people out of n:
math.comb(20, 6)= 38,760.
Binomial PMF:
P(X = k) = comb(n, k) × pᵏ × (1 − p)ⁿ⁻ᵏ
The number of successes in n independent Bernoulli trials with chance p follows the binomial distribution, Binomial(n, p). A function giving the probability of each exact value of a discrete random variable is a probability mass function (PMF). "Mass": each value carries its own lump of probability, and all lumps add to 1.
from math import comb
import numpy as np
def binom_pmf(k, n, p): # ways x chance of one way
return comb(n, k) * p**k * (1 - p)**(n - k)
n, p = 20, 0.3
print("ways to pick 6 buyers of 20:", comb(20, 6))
print("P(X = 6) =", round(binom_pmf(6, n, p), 4))
print("sum of all P:", round(sum(binom_pmf(k, n, p) for k in range(n + 1)), 6))
print("100 people, P(X = 30) =", round(binom_pmf(30, 100, p), 4))
rng = np.random.default_rng(10)
sales = rng.binomial(n, p, size=100_000) # 100,000 simulated days
print("simulated P(X = 6) =", round(np.mean(sales == 6), 4))
Output:
ways to pick 6 buyers of 20: 38760
P(X = 6) = 0.1916
sum of all P: 1.0
100 people, P(X = 30) = 0.0868
simulated P(X = 6) = 0.1938
rng.binomial(n, p, size=100_000) simulates 100,000 days at once; np.mean(sales == 6) is the fraction of days with exactly 6 cups.
The shape of the PMF (n = 20, p = 0.3)
| Cups k | 2 | 4 | 6 | 8 | 10 |
|---|---|---|---|---|---|
| P(X = k) | 0.028 | 0.130 | 0.192 | 0.114 | 0.031 |
The peak is at n × p = 6.
Poisson: counting arrivals
New question: how many customers arrive in the next 10 minutes? There's no fixed number of tries n, only an average rate: λ (lambda) = 4 per 10 minutes.
Trick: slice the 10 minutes into many tiny slices, so short that at most one customer arrives in each. Each slice is a Bernoulli trial with p = λ / slices. As the slices get thinner, the binomial answer settles:
from math import comb, exp, factorial
import numpy as np
lam = 4 # 4 customers per 10 minutes, on average
def poisson_pmf(k):
return lam**k * exp(-lam) / factorial(k)
for slices in [10, 100, 10_000]: # binomial: split 10 minutes into tiny slices
p = lam / slices
binom6 = comb(slices, 6) * p**6 * (1 - p)**(slices - 6)
print(slices, "slices: P(X = 6) =", round(binom6, 4))
print("Poisson: P(X = 6) =", round(poisson_pmf(6), 4))
rng = np.random.default_rng(3)
arrivals = rng.poisson(lam, size=100_000)
print("simulated: P(X = 6) =", round(np.mean(arrivals == 6), 4), " average:", arrivals.mean())
Output:
10 slices: P(X = 6) = 0.1115
100 slices: P(X = 6) = 0.1052
10000 slices: P(X = 6) = 0.1042
Poisson: P(X = 6) = 0.1042
simulated: P(X = 6) = 0.1043 average: 4.01382
The limit has its own formula, the Poisson PMF:
P(X = k) = λᵏ e^(−λ) / k!
(e ≈ 2.718 is Euler's number; k! = 1 × 2 × … × k is "k factorial".) Use the Poisson distribution for counts of events in a stretch of time or space: customers per hour, typos per page, requests per second.
Continuous variables: P(exactly 170) = 0
import numpy as np
rng = np.random.default_rng(5)
heights = rng.normal(165, 7, size=100_000) # made-up heights in cm
print(heights[:4])
print("P(height == 170) =", np.mean(heights == 170))
print("P(169.5 < height < 170.5) =", np.mean((heights > 169.5) & (heights < 170.5)))
print("P(169.95 < height < 170.05) =", np.mean((heights > 169.95) & (heights < 170.05)))
Output:
[159.38648002 155.72948703 163.26146865 167.94311667]
P(height == 170) = 0.0
P(169.5 < height < 170.5) = 0.04586
P(169.95 < height < 170.05) = 0.00444
Not one of 100,000 people is exactly 170 cm: with infinitely many possible values, every single exact value has probability 0. So a PMF is useless here. Ask about a range instead:
| Strip around 170 | width | probability | probability ÷ width |
|---|---|---|---|
| 169.5 to 170.5 | 1 cm | 0.0459 | ≈ 0.046 per cm |
| 169.95 to 170.05 | 0.1 cm | 0.0044 | ≈ 0.044 per cm |
The chance per centimetre stays steady. That is a density.
The probability density function (PDF)
Analogy: crowd density on a railway platform. You can't count people at a point, but you can say "5 people per metre near the stairs". People on a stretch = density × length.
- The PDF f(x) is the probability per unit near x. It's a density, not a probability.
- P(a < X < b) = area under f between a and b (an integral: adding up many thin strips).
- Total area = 1, and P(X = exactly x) = 0.
- A density can be greater than 1 (a packed spot on the platform); only areas must be ≤ 1.
The normal PDF with centre μ (mu, the mean) and spread σ (sigma, the standard deviation):
f(x) = e^(−(x − μ)² / (2σ²)) / (σ √(2π))
import numpy as np
def normal_pdf(x, mu, sigma):
return np.exp(-(x - mu)**2 / (2 * sigma**2)) / (sigma * np.sqrt(2 * np.pi))
print("density at 170:", round(normal_pdf(170, 165, 7), 4))
print("area of a 1 cm strip:", round(normal_pdf(170, 165, 7) * 1, 4))
xs = np.linspace(100, 230, 100_001) # tiny strips
print("total area:", round(np.trapezoid(normal_pdf(xs, 165, 7), xs), 4))
print("uniform(0, 0.5) density:", 1 / 0.5)
Output:
density at 170: 0.0442
area of a 1 cm strip: 0.0442
total area: 1.0
uniform(0, 0.5) density: 2.0
- The density at 170 (0.0442) matches the ≈ 0.045 per cm we measured.
np.linspace(100, 230, 100_001)makes 100,001 evenly spaced points;np.trapezoid(y, x)adds up the thin trapezoid strips under the curve: the area is 1.0.- A uniform distribution on 0 to 0.5 (every value equally likely) has density 1 / 0.5 = 2: bigger than 1 and perfectly valid.
The cumulative distribution function (CDF)
Most real questions are "at most": will I wait at most 3 minutes? Sell at most 6 cups?
- CDF: F(x) = P(X ≤ x), all the probability up to x. Cumulative means piling up.
- Discrete: add PMF values up to x. Continuous: area under the PDF up to x.
- P(a < X ≤ b) = F(b) − F(a). F climbs from 0 to 1.
import numpy as np
from scipy.stats import norm, binom
rng = np.random.default_rng(8)
wait = rng.uniform(0, 10, size=100_000) # bus every 10 min: wait 0 to 10
print("P(wait <= 3): simulated", round(np.mean(wait <= 3), 4), " CDF", 3 / 10)
print("P(height <= 170) =", round(norm.cdf(170, loc=165, scale=7), 4))
print("P(160 < height <= 170) =", round(norm.cdf(170, 165, 7) - norm.cdf(160, 165, 7), 4))
print("P(cups <= 6) =", round(binom.cdf(6, 20, 0.3), 4))
Output:
P(wait <= 3): simulated 0.2996 CDF 0.3
P(height <= 170) = 0.7625
P(160 < height <= 170) = 0.5249
P(cups <= 6) = 0.608
- Bus every 10 minutes, you arrive at a random moment: the wait is Uniform(0, 10), so F(3) = 3/10.
- SciPy (
scipy.stats) has ready-made distributions:norm.cdf(x, loc=μ, scale=σ),binom.cdf(k, n, p),poisson.cdf(k, λ), and the matching.pmf,.pdf,.sf(survival function, 1 − CDF) and.ppf(inverse CDF).
The normal (Gaussian) distribution
- A symmetric bell centred at the mean μ with spread σ; also called the Gaussian (after Gauss).
- It appears when many small independent effects add up (height from many genes and meals; measurement errors).
- z-score: z = (x − μ) / σ, how many σ a value is from the mean.
- The standard normal has μ = 0, σ = 1; every normal becomes standard after converting to z-scores.
- 68–95–99.7 rule: about 68% of values within 1σ, 95% within 2σ, 99.7% within 3σ.
import numpy as np
rng = np.random.default_rng(1)
heights = rng.normal(165, 7, size=100_000)
z = (heights - 165) / 7 # standardise: how many sigmas away
print("z mean:", round(z.mean(), 3), " z std:", round(z.std(), 3))
for k in [1, 2, 3]:
print(f"within {k} sigma:", round(np.mean(np.abs(z) < k), 4))
dice = rng.integers(1, 7, size=(100_000, 10)).sum(axis=1) # add 10 dice
counts = [int(np.sum(dice == s)) for s in range(25, 50, 5)]
print("dice counts:", counts) # sums 25, 30, 35, 40, 45
Output:
z mean: -0.005 z std: 0.997
within 1 sigma: 0.6841
within 2 sigma: 0.955
within 3 sigma: 0.9971
dice counts: [1288, 4853, 7285, 4835, 1298]
The sum of 10 flat dice is already bell-shaped: low counts at 25 and 45, the peak at 35. Episode 11 shows why (the central limit theorem).
Expectation, variance and standard deviation
- Expectation (expected value) E[X] = Σ x · P(x): the long-run average. For a die, 3.5, a value you can never roll.
- The obvious spread measure, the average distance from the mean, is always 0: values below the mean cancel those above (the same trap as the plain average error in Episode 8).
- Variance Var(X) = Σ (x − E[X])² · P(x): average squared distance. Its units are squared.
- Standard deviation σ = √Var(X): spread in the original units.
import numpy as np
faces = np.array([1, 2, 3, 4, 5, 6])
probs = np.full(6, 1 / 6)
E = np.sum(faces * probs) # expectation
print("expected value:", E)
print("average distance from E:", np.sum((faces - E) * probs))
var = np.sum((faces - E)**2 * probs) # variance
print("variance:", round(var, 4), " std dev:", round(np.sqrt(var), 4))
rng = np.random.default_rng(2)
rolls = rng.integers(1, 7, size=100_000)
print("simulated mean:", rolls.mean(), " var:", round(rolls.var(), 4))
Output:
expected value: 3.5
average distance from E: 0.0
variance: 2.9167 std dev: 1.7078
simulated mean: 3.49934 var: 2.9207
Mean and variance of each distribution
| Distribution | Parameters | Mean | Variance | Example |
|---|---|---|---|---|
| Bernoulli | p | p | p(1 − p) | one passer-by |
| Binomial | n, p | np | np(1 − p) | cups from 20 passers-by |
| Poisson | λ | λ | λ | customers per 10 min |
| Uniform | a, b | (a + b)/2 | (b − a)²/12 | bus wait 0 to 10 |
| Normal | μ, σ | μ | σ² | heights |
Sampling with NumPy
import numpy as np
rng = np.random.default_rng(42)
N = 100_000
samples = {
"bernoulli p=0.3": rng.binomial(1, 0.3, N),
"binomial 20, 0.3": rng.binomial(20, 0.3, N),
"poisson 4": rng.poisson(4, N),
"uniform 0 to 10": rng.uniform(0, 10, N),
"normal 165, 7": rng.normal(165, 7, N),
}
for name, x in samples.items():
print(f"{name:17} mean {x.mean():8.3f} variance {x.var():7.3f}")
Output:
bernoulli p=0.3 mean 0.300 variance 0.210
binomial 20, 0.3 mean 5.997 variance 4.205
poisson 4 mean 4.001 variance 3.963
uniform 0 to 10 mean 5.007 variance 8.365
normal 165, 7 mean 164.986 variance 49.406
Every sampled mean and variance matches the table: 0.3/0.21, 6/4.2, 4/4, 5/8.33, 165/49. Sampling means drawing random values from a distribution; it checks any formula.
NumPy (rng = np.random.default_rng(seed)) |
Draws from |
|---|---|
rng.random(size) |
Uniform(0, 1) |
rng.integers(low, high, size) |
whole numbers low … high − 1 |
rng.binomial(n, p, size) |
Binomial (n = 1 → Bernoulli) |
rng.poisson(lam, size) |
Poisson |
rng.uniform(a, b, size) |
Uniform(a, b) |
rng.normal(loc, scale, size) |
Normal(μ = loc, σ = scale) |
Common mistakes
import numpy as np
rng = np.random.default_rng(0)
print(rng.normal(165, -7, size=3))
Output:
ValueError: scale < 0
| Mistake | What happens | Why | Fix |
|---|---|---|---|
rng.normal(165, -7) |
ValueError: scale < 0 |
σ is a distance, never negative | use a positive σ |
rng.binomial(20, 1.3) |
ValueError: p < 0, p > 1 or p is NaN |
p is a probability | keep 0 ≤ p ≤ 1 |
rng.poisson(-4) |
ValueError: lam < 0 or lam is NaN |
an average count can't be negative | λ ≥ 0 |
rng.normal(165, 49) meaning variance 49 |
variance comes out 2,401 | scale is σ, not σ² |
rng.normal(165, 7) |
rng.integers(1, 6) for a die |
never rolls a 6 | the upper end is excluded | rng.integers(1, 7) |
| P(height == 170) from a PDF | always 0 | single points have no area | ask for a range: F(b) − F(a) |
| "density 2.0 is impossible" | false alarm | density is per unit | only areas must be ≤ 1 |
| no seed in a simulation | different numbers every run | no fixed starting point | np.random.default_rng(seed) |
Mini project: how much milk should Raju plan for?
20 people pass per hour and 3 in 10 buy. Find the smallest number of cups k that covers at least 95% of hours, using the binomial CDF, then check it by simulation.
{{PY:/tmp/claude-0/s/m10.py}}
binom.ppf(0.95, n, p) is the inverse CDF (percent point function): the smallest k with F(k) ≥ 0.95. Planning for 9 cups an hour, Raju runs out in only about 5% of hours.
Practice
- P(at least 8 cups) for n = 20, p = 0.3.
Solution
"At least 8" = 1 − P(X ≤ 7).
{{PY:/tmp/claude-0/s/h10a.py}}
- With λ = 4 customers per 10 minutes, P(more than 6 customers)?
Solution
P(X > 6) = 1 − F(6), which SciPy calls the survival function sf.
{{PY:/tmp/claude-0/s/h10b.py}}
- Does the 68% rule hold for a uniform distribution?
Solution
No. For Uniform(0, 10), σ = 10/√12 ≈ 2.887, and the fraction within 1σ is 2 × 2.887 / 10 ≈ 0.577. The 68–95–99.7 rule is only for normal distributions.
{{PY:/tmp/claude-0/s/h10c.py}}
- Mean and variance of a loaded die that lands six half the time (the other faces 0.1 each).
Solution
E = 0.1 × (1 + 2 + 3 + 4 + 5) + 0.5 × 6 = 1.5 + 3 = 4.5.
{{PY:/tmp/claude-0/s/h10d.py}}
Quick quiz
from math import comb
n, p = 4, 0.5
pmf = [comb(n, k) * p**k * (1 - p)**(n - k) for k in range(n + 1)]
print(pmf[2], sum(pmf[:2]))
A) 0.375 0.3125 B) 0.5 0.3125 C) 0.375 0.25 D) 0.25 0.375
Answer
A) 0.375 0.3125. comb(4, 2) = 6 patterns × 0.5⁴ = 6/16 = 0.375. P(X ≤ 1) = (1 + 4)/16 = 0.3125, the CDF at 1.
Interview questions
- PMF vs PDF? A PMF gives the probability of each exact value of a discrete variable; a PDF gives a density for a continuous variable, and probabilities are areas under it.
- Can a PDF be greater than 1? Yes. Density is per unit; only the total area (and every area) must be ≤ 1. Uniform(0, 0.5) has density 2.
- What does the CDF give, and how do you get P(a < X ≤ b)? F(x) = P(X ≤ x); F(b) − F(a).
- When do you use binomial vs Poisson? Binomial: a fixed number n of independent yes/no trials. Poisson: counts of events over an interval with a known average rate λ; it is the limit of the binomial with large n and small p.
- Mean and variance of Binomial(n, p) and Poisson(λ)? np and np(1 − p); λ and λ.
- Why is the normal distribution so common? Many small independent effects add up to it (central limit theorem, Episode 11).
- What is a z-score? (x − μ)/σ: how many standard deviations from the mean; it turns any normal into the standard normal.
- Why do we square deviations for variance? Plain deviations from the mean always average to zero; squaring makes every deviation count positively.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Random variable | A number that depends on chance | cups sold tomorrow |
| Probability distribution | The probability of every value of a random variable | P(0 cups), …, P(20 cups) |
| Discrete random variable | Countable values with gaps | cups: 0, 1, 2 |
| Continuous random variable | Any value in a range | height 165.37 cm |
| Bernoulli trial | One try with two outcomes: success (1) or failure (0) | one passer-by buys or not |
| Bernoulli distribution | Distribution of one Bernoulli trial; parameter p | P(1) = 0.3 |
| Success / failure | The two outcomes of a trial, coded 1 / 0 | buy / no buy |
Combinations, math.comb(n, k) |
Number of ways to choose k items out of n | comb(20, 6) = 38,760 |
| Binomial distribution | Number of successes in n independent Bernoulli trials | cups from 20 people |
| Probability mass function (PMF) | Probability of each exact value (discrete) | P(X = 6) = 0.1916 |
| Poisson distribution | Counts of events in an interval with average rate λ | customers per 10 min |
| λ (lambda) | Average count per interval (Poisson) | 4 |
| Factorial, k! | 1 × 2 × … × k | 3! = 6 |
| Euler's number e | ≈ 2.718, the base of natural logarithms | e^(−4) |
| Density | Amount per unit (of length, time …) | 0.044 per cm |
| Probability density function (PDF) | Density of a continuous variable; area = probability | normal PDF at 170 = 0.0442 |
| Integral (area under a curve) | Sum of many thin strips under a curve | np.trapezoid |
| Cumulative distribution function (CDF) | F(x) = P(X ≤ x) | F(170) = 0.7625 |
Survival function (sf) |
1 − CDF = P(X > x) | P(X > 6) = 0.1107 |
Inverse CDF (ppf) |
Smallest x with F(x) ≥ q | 9 cups for 95% |
| Uniform distribution | Every value in [a, b] equally likely | bus wait 0 to 10 |
| Normal (Gaussian) distribution | Symmetric bell with mean μ and standard deviation σ | heights |
| μ (mu), σ (sigma) | Mean and standard deviation | 165, 7 |
| Standard normal | Normal with μ = 0, σ = 1 | z-scores |
| z-score | (x − μ)/σ | 172 cm → z = 1 |
| 68–95–99.7 rule | Share of a normal within 1, 2, 3 σ | 0.6841, 0.955, 0.9971 |
| Expectation, E[X] | Σ x · P(x): long-run average | die: 3.5 |
| Variance | Average squared distance from the mean | die: 2.9167 |
| Standard deviation | √variance, in original units | die: 1.7078 |
| Sampling | Drawing random values from a distribution | rng.normal(165, 7, N) |
| Random number generator, seed | Produces pseudo-random numbers; a seed makes them repeat | default_rng(42) |
SciPy, scipy.stats |
Python library with ready-made distributions | norm.cdf |
Summary
- A distribution gives the chance of every value of a random variable.
- Discrete variables have a PMF; continuous variables have a PDF where probability = area, and any exact value has probability 0.
- The CDF F(x) = P(X ≤ x) answers every "at most" question; ranges are F(b) − F(a).
- Bernoulli (one try), binomial (successes in n tries), Poisson (counts at rate λ), uniform (all equal), normal (the bell; z-scores; 68–95–99.7).
- Expectation is the long-run average; variance is the average squared distance; standard deviation is its square root.
- NumPy's
default_rngsamples every distribution; sampling checks any formula.
Next episode: Maths for ML 9, statistics for ML: mean, median, percentiles, IQR, skewness, correlation, the central limit theorem, confidence intervals, p-values, t-tests and chi-square tests.