SyntaxPilot
Courses/Machine Learning/Lesson 11

Machine Learning · Lesson 11 of 90 · 15 min

Maths 9: statistics for ML

meanmedianmodepercentilesquartilesinterquartile range (IQR)skewnesscovariancePearson correlationSpearman correlationcorrelation vs causationpopulation vs samplecentral limit theoremconfidence interval

Machine Learning Full Course · Episode 11. Every code sample below was run with Python 3.13, NumPy 2.5.3 and SciPy 1.18.1; the output shown is the real output. All the data (salaries, delivery times, temperatures, cups, rain, heights) is made up for teaching.

Probability vs statistics. Probability (Episodes 9 and 10) goes from a known distribution to data: "if the coin is fair, how likely are 7 heads?" Statistics goes the other way, from data to the truth: "I saw 7 heads; is the coin fair?" Every time you summarise a dataset, compare two models or ask "is this improvement real or luck?", you're doing statistics.

The job offer: mean, median, mode

A 10-person startup says: "our average salary is 17.9 lakh a year." Almost everyone you ask earns 5 to 9 lakh. The founder pays himself 120.

import statistics as st

salaries = [6, 5, 6, 7, 6, 8, 9, 5, 7, 120]   # lakh a year; last = founder
print("mean:  ", st.mean(salaries))
print("median:", st.median(salaries))
print("mode:  ", st.mode(salaries))
print("mean without the founder:", round(st.mean(salaries[:-1]), 2))

Output:

mean:   17.9
median: 6.5
mode:   6
mean without the founder: 6.56
Measure Definition Here Moved by the founder?
Mean sum ÷ count 17.9 yes, a lot (6.56 without him)
Median middle value after sorting (average of the two middle values for an even count) 6.5 no
Mode most common value 6 (three people) no

An extreme value far from the rest is an outlier. The mean feels outliers; the median doesn't. For skewed money data (salaries, house prices), report the median.

Spread: percentiles, quartiles and the IQR

The obvious spread measure, the range (max − min), is wrecked by one outlier:

import numpy as np

rng = np.random.default_rng(11)
times = np.round(rng.gamma(9, 3, size=1000), 1)    # delivery times, minutes
print("range:", times.min(), "to", times.max())
q1, q2, q3 = np.percentile(times, [25, 50, 75])
print("Q1:", q1, " median:", q2, " Q3:", q3, " IQR:", round(q3 - q1, 1))
print("90th percentile:", np.percentile(times, 90))

times[0] = 300                                    # one order stuck for 5 hours
q1, q3 = np.percentile(times, [25, 75])
print("after outlier -> range:", times.max() - times.min(), " IQR:", round(q3 - q1, 1))

Output:

range: 7.0 to 60.0
Q1: 20.6  median: 26.8  Q3: 32.725  IQR: 12.1
90th percentile: 39.5
after outlier -> range: 293.0  IQR: 12.2
  • The p-th percentile is the value p% of the data lies below. 90th percentile = 39.5 minutes: "90% of orders arrive within 39.5 minutes".
  • The quartiles cut the data into four quarters: Q1 = 25th percentile, Q2 = median, Q3 = 75th percentile.
  • The interquartile range, IQR = Q3 − Q1, is the spread of the middle half: 12.1 before the stuck order, 12.2 after. The range jumped from 53 to 293.
  • A statistic that outliers barely move is robust: median and IQR are robust; mean and range are not.
  • np.percentile(x, [25, 50, 75]) takes percentages from 0 to 100.

Skewness

import numpy as np
from scipy.stats import skew

rng = np.random.default_rng(4)
heights = rng.normal(165, 7, 10_000)
salaries = rng.lognormal(2, 0.6, 10_000)        # a long right tail
for name, x in [("heights", heights), ("salaries", salaries)]:
    med = np.median(x)
    print(f"{name:8}  mean {x.mean():7.2f}  median {med:7.2f}  skew {skew(x):5.2f}")

Output:

heights   mean  165.11  median  165.13  skew -0.04
salaries  mean    8.87  median    7.41  skew  2.11
  • Skewness measures lopsidedness. About 0 = symmetric (mean ≈ median).
  • Positive (right) skew: a long tail of big values pulls the mean above the median (salaries: 8.87 vs 7.41, skew 2.11).
  • Negative (left) skew: a long tail of small values (for example, exam scores on an easy test).
  • rng.lognormal draws values whose logarithm is normal, a common model for incomes.

Covariance: the obvious measure fails

Raju thinks hot afternoons sell less chai. For each day, multiply (temperature − mean temperature) × (cups − mean cups) and average: hot day with few cups gives (+) × (−) = negative. That average is the covariance.

import numpy as np

temp_c = np.array([14, 18, 22, 25, 28, 31, 35, 38])     # afternoon temperature
cups = np.array([152, 140, 131, 118, 112, 100, 86, 80])   # chai sold that day

cov = np.mean((temp_c - temp_c.mean()) * (cups - cups.mean()))
print("covariance (Celsius):   ", round(cov, 2))
temp_f = temp_c * 9 / 5 + 32
cov_f = np.mean((temp_f - temp_f.mean()) * (cups - cups.mean()))
print("covariance (Fahrenheit):", round(cov_f, 2))

r = cov / (temp_c.std() * cups.std())                     # Pearson correlation
print("Pearson r (Celsius):", round(r, 4))
print("Pearson r (Fahrenheit):", round(np.corrcoef(temp_f, cups)[0, 1], 4))

Output:

covariance (Celsius):    -184.08
covariance (Fahrenheit): -331.34
Pearson r (Celsius): -0.9977
Pearson r (Fahrenheit): -0.9977

Covariance is −184 in Celsius but −331 in Fahrenheit: same data, different number. Covariance depends on units, so its size means nothing on its own; only its sign (direction) does.

The fix: Pearson correlation. Divide by both standard deviations, which cancels the units:

r = cov(x, y) / (σx σy)

  • r is always between −1 and +1.
  • +1: points on a rising straight line; −1: a falling straight line; 0: no straight-line (linear) relationship.
  • np.corrcoef(x, y)[0, 1] computes it (the function returns a 2 × 2 matrix of all pairwise correlations).

Spearman rank correlation

Pearson only measures straight-line relationships. Marks always rise with study hours, but along a curve:

from scipy.stats import pearsonr, spearmanr, rankdata

hours = [1, 2, 3, 4, 5, 6, 7, 8]                  # hours studied
score = [20, 24, 30, 40, 55, 75, 99, 100]         # always rises, but curves
print("Pearson: ", round(pearsonr(hours, score).statistic, 4))
print("ranks of score:", rankdata(score))
r_ranks = pearsonr(rankdata(hours), rankdata(score)).statistic
print("Pearson on ranks:", round(r_ranks, 4))
print("Spearman:", round(spearmanr(hours, score).statistic, 4))

Output:

Pearson:  0.9734
ranks of score: [1. 2. 3. 4. 5. 6. 7. 8.]
Pearson on ranks: 1.0
Spearman: 1.0
  • The rank of a value is its position in sorted order (scipy.stats.rankdata).
  • Spearman's rank correlation = Pearson's r computed on the ranks. It measures whether one variable always rises (or always falls) with the other, curved or not (a monotonic relationship). Here it's exactly 1.0, while Pearson gives 0.9734.
  • Because it uses ranks, an extreme value can only move one rank, so Spearman is more robust to outliers.

Correlation is not causation

On days the shop next door sells many umbrellas, Raju sells more chai. But umbrellas don't make anyone thirsty: rain drives both. A hidden variable that causes both is a confounder.

import numpy as np

rng = np.random.default_rng(9)
rain = rng.uniform(0, 50, 200)                        # mm of rain, 200 days
umbrellas = 2 * rain + rng.normal(0, 10, 200)         # rain -> umbrellas sold
chai = 100 + 1.5 * rain + rng.normal(0, 10, 200)      # rain -> chai sold
print("corr(umbrellas, chai):", round(np.corrcoef(umbrellas, chai)[0, 1], 3))

dry = rain < 5                                        # only the dry days
print("on dry days only:", round(np.corrcoef(umbrellas[dry], chai[dry])[0, 1], 3))

Output:

corr(umbrellas, chai): 0.844
on dry days only: -0.03

Holding the confounder fixed (dry days only) makes the correlation vanish (0.844 → −0.03). Models learn correlations: fine for prediction, wrong for deciding what to change. Proving causes needs experiments (like the A/B test below).

Population vs sample

Term Meaning Example
Population every member we care about all adults in a city
Sample the members we actually measure 5 people
Parameter a number describing the population (usually unknown) true mean μ, true variance σ²
Statistic the same number computed from a sample (changes every sample) sample mean x̄, sample variance
import numpy as np

rng = np.random.default_rng(0)
population = rng.normal(165, 7, 1_000_000)        # every adult in a city
print("population variance:", round(population.var(), 1))

for _ in range(3):
    sample = rng.choice(population, 5)            # we can only measure 5
    print("sample variance:", round(sample.var(), 1))

v0 = [rng.choice(population, 5).var() for _ in range(20_000)]
v1 = [rng.choice(population, 5).var(ddof=1) for _ in range(20_000)]
print("avg sample variance, divide by n:  ", round(np.mean(v0), 1))
print("avg sample variance, divide by n-1:", round(np.mean(v1), 1))

Output:

population variance: 49.1
sample variance: 27.1
sample variance: 85.0
sample variance: 36.0
avg sample variance, divide by n:   39.2
avg sample variance, divide by n-1: 49.4
  • Every sample gives a different variance (27.1, 85.0, 36.0; true 49.1).
  • Dividing by n gives too small an answer on average (39.2): the sample's own mean sits closer to its points than the true mean does. This systematic error is called bias.
  • Dividing by n − 1 (Bessel's correction) fixes it (49.4). In NumPy: x.var(ddof=1), x.std(ddof=1). pandas uses n − 1 by default; NumPy uses n by default.

The central limit theorem (CLT)

import numpy as np
from scipy.stats import skew

rng = np.random.default_rng(6)
def show(label, x):
    print(f"{label}: mean {x.mean():.2f}  std {x.std():.2f}  skew {skew(x):.2f}")

waits = rng.exponential(10, 1_000_000)        # waiting times: very lopsided
show("one wait  ", waits)
means = rng.choice(waits, size=(20_000, 50)).mean(axis=1)   # 20,000 x 50
show("mean of 50", means)
print("sigma / sqrt(n) =", round(waits.std() / np.sqrt(50), 2))
near = np.abs(means - means.mean()) < 2 * means.std()
print("within 2 std:", round(np.mean(near), 3))

Output:

one wait  : mean 10.00  std 9.98  skew 1.99
mean of 50: mean 10.00  std 1.41  skew 0.27
sigma / sqrt(n) = 1.41
within 2 std: 0.955
  • Single waiting times are lopsided (skew 1.99). Means of 50 waits are nearly symmetric (skew 0.27), and 95.5% of them lie within 2 standard deviations, like a normal.
  • Central limit theorem: the average of many independent values from almost any distribution (with finite variance) is approximately normal. That is why the bell curve is everywhere, and why 10 dice made a bell in Episode 10.
  • Standard error (SE): the standard deviation of a sample mean = σ / √n (here 9.98 / √50 = 1.41). Four times the data halves the wobble.

Confidence intervals

A 95% confidence interval (CI) for a mean: sample mean ± 1.96 × SE (1.96 because 95% of a normal lies within 1.96 σ of its centre).

import numpy as np

rng = np.random.default_rng(12)
true_mean = 165                                   # unknown in real life
hits = 0
for i in range(10_000):
    sample = rng.normal(true_mean, 7, 100)        # one survey of 100 people
    se = sample.std(ddof=1) / np.sqrt(100)        # standard error
    low, high = sample.mean() - 1.96 * se, sample.mean() + 1.96 * se
    if i < 3:
        print(f"survey {i + 1}: 95% CI {low:.2f} to {high:.2f}")
    hits += low <= true_mean <= high
print("intervals that caught the true mean:", hits / 10_000)

Output:

survey 1: 95% CI 164.05 to 166.57
survey 2: 95% CI 163.44 to 166.12
survey 3: 95% CI 164.39 to 167.15
intervals that caught the true mean: 0.9441

About 95% of the intervals catch the true mean (94.4% here). That is the meaning of "95% confidence": the method catches the truth 95 times in 100. It does not mean "there's a 95% chance the true mean is in this particular interval".

Hypothesis tests: is masala chai really better?

Raju runs 30 days of the old recipe and 30 days of masala chai: an A/B test (comparing two versions). Masala sells about 7 more cups a day. The obvious rule "bigger average wins" fails, because luck alone creates gaps:

import numpy as np

rng = np.random.default_rng(22)
a = rng.normal(100, 12, 30)        # 30 days, old recipe
b = rng.normal(100, 12, 30)        # 30 more days, SAME recipe
print("first month: ", round(a.mean(), 1), "cups a day")
print("second month:", round(b.mean(), 1), "cups a day")
print("gap:", round(b.mean() - a.mean(), 1), "cups, with nothing changed")

Output:

first month:  98.7 cups a day
second month: 104.5 cups a day
gap: 5.8 cups, with nothing changed

A 5.8-cup gap with nothing changed. So we need to know how big a gap luck can make.

The hypothesis test, step by step:

  1. Null hypothesis H₀: "no real difference; any gap is luck". (Null = nothing.)
  2. Alternative hypothesis H₁: "there is a real difference".
  3. p-value: if H₀ were true, the probability of a gap at least as big as the one observed.
  4. If p < the significance level α (usually 0.05), reject H₀: luck is a poor explanation. Otherwise, fail to reject H₀ (not the same as proving it).
  5. The p-value is not P(H₀ is true). It's P(data this extreme | H₀): conditionals don't flip (Episode 9).

Computing a p-value by shuffling (a permutation test): if the recipe doesn't matter, the labels "old" and "new" are meaningless. Shuffle all 60 days, split into two fake groups, and count how often the fake gap is as big as the real one.

import numpy as np
from scipy.stats import ttest_ind

def shuffle_p(x, y, rng):              # how often does pure luck give this gap?
    gap = y.mean() - x.mean()
    both = np.concatenate([x, y])
    count = 0
    for _ in range(10_000):
        rng.shuffle(both)              # if the null is true, labels don't matter
        count += abs(both[30:].mean() - both[:30].mean()) >= abs(gap)
    return float(round(gap, 2)), float(count / 10_000)
rng = np.random.default_rng(22)
a, b = rng.normal(100, 12, 30), rng.normal(100, 12, 30)   # same recipe twice
print("same recipe: gap, p =", shuffle_p(a, b, rng))
rng = np.random.default_rng(5)
old, new = rng.normal(100, 12, 30), rng.normal(107, 12, 30)
print("masala chai: gap, p =", shuffle_p(old, new, rng))
print("t-test p-values:", round(ttest_ind(b, a).pvalue, 4), round(ttest_ind(new, old).pvalue, 4))

Output:

same recipe: gap, p = (5.85, 0.1081)
masala chai: gap, p = (7.04, 0.0234)
t-test p-values: 0.1161 0.0212
Comparison Gap Shuffle p t-test p Decision at α = 0.05
same recipe twice 5.85 0.1081 0.1161 keep H₀ (luck explains it)
masala vs old 7.04 0.0234 0.0212 reject H₀ (effect looks real)

The t-test (scipy.stats.ttest_ind, "independent samples") compares two means by measuring the gap in standard errors (the t statistic) and uses the t distribution to get the p-value. It agrees closely with shuffling.

Two kinds of wrong decision (more in Episode 24): rejecting H₀ when it's true (a false positive, type I error, happens α of the time) and keeping H₀ when it's false (a false negative, type II error).

The chi-square test (for categories)

The t-test compares averages of numbers. For categories (tea or coffee, morning or evening), use the chi-square (χ²) test of independence. Raju's order book from Episode 9:

from scipy.stats import chi2_contingency

#            tea  coffee
table = [[120, 30],          # morning
         [60, 40]]           # evening
res = chi2_contingency(table)
print("chi-square:", round(res.statistic, 2), " p-value:", round(res.pvalue, 4))
print("expected if independent:")
print(res.expected_freq)

Output:

chi-square: 10.93  p-value: 0.0009
expected if independent:
[[108.  42.]
 [ 72.  28.]]
  • Expected counts if time and drink were independent: row total × column total ÷ grand total (e.g. 150 × 180 / 250 = 108 morning teas).
  • χ² statistic = Σ (observed − expected)² / expected: how far reality is from independence. For 2 × 2 tables SciPy applies Yates' continuity correction by default (a small adjustment), giving 10.93.
  • p = 0.0009 < 0.05: drink choice and time of day are not independent, matching Episode 9.

Common mistakes

from scipy.stats import pearsonr

temp = [20, 25, 30, 35]
cups = [100, 100, 100, 100]        # a week when sales never changed
print(pearsonr(temp, cups))

Output:

PearsonRResult(statistic=np.float64(nan), pvalue=np.float64(nan))
main.py:5: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
Mistake What happens Why Fix
correlation of a constant column ConstantInputWarning, result nan divides by a standard deviation of 0 check columns vary
reporting the mean of skewed data "17.9 lakh average" outliers drag the mean report the median (6.5)
comparing covariances −184 vs −331 for the same data covariance has units use correlation r
np.percentile(x, 0.9) for the 90th a tiny value NumPy wants 0 to 100 np.percentile(x, 90)
np.std(x) for a sample slightly too small NumPy divides by n np.std(x, ddof=1)
correlation ⇒ causation umbrellas "cause" chai confounder (rain) run an experiment
"p = 0.02 means a 2% chance H₀ is true" wrong meaning p = P(data this extreme | H₀) read it as a luck check
"p = 0.06 proves no effect" wrong conclusion not rejecting ≠ proving H₀ collect more data
pearsonr with lists of different lengths ValueError: `x` and `y` must be broadcastable. pairs must match same length

Mini project: a bootstrap confidence interval and the IQR outlier fence

The 1.96 × SE formula works for means. For a median there's no simple formula, so use the bootstrap: resample the data with replacement many times, compute the median each time, and take the middle 95% of those medians. Then flag outliers with the box-plot rule: above Q3 + 1.5 × IQR (Episodes 13 and 14 use this).

{{PY:/tmp/claude-0/s/m11.py}}

Practice

  1. Median and IQR of [3, 7, 8, 5, 12, 14, 21, 13, 18].
Solution

{{PY:/tmp/claude-0/s/h11a.py}}

  1. Pearson vs Spearman for y = x³, x = 1 … 10.
Solution

y always rises with x, so Spearman is exactly 1; the curve isn't a straight line, so Pearson is below 1.

{{PY:/tmp/claude-0/s/h11b.py}}

  1. A shuffle test with a smaller true effect (3 cups): what p-value do you get?
Solution

{{PY:/tmp/claude-0/s/h11c.py}}

p ≈ 0.31: with 30 days per group and daily noise of 12 cups, a 3-cup effect can't be told apart from luck. More days would shrink the standard error.

  1. Is p = 0.06 proof that the new recipe has no effect?
Solution

No. p = 0.06 means luck alone would produce a gap this big about 6% of the time. We fail to reject H₀ at α = 0.05, but that is "not enough evidence", not "proof of no effect". A larger experiment might detect it.

Quick quiz

import statistics as st

x = [2, 4, 4, 5, 100]
print(st.mean(x), st.median(x), st.mode(x))

A) 23 4 4 B) 4 4 4 C) 23 4 5 D) 23.0 4.5 4

Answer

A) 23 4 4. Mean = 115 / 5 = 23 (the outlier 100 drags it), median = 4, mode = 4.

Interview questions

  1. Mean vs median: when do you use which? Median for skewed data or data with outliers (incomes, prices, latencies); mean when the data is roughly symmetric or when totals matter.
  2. What is the IQR and why is it robust? Q3 − Q1, the spread of the middle 50%; the extremes never enter it.
  3. Covariance vs correlation? Covariance depends on units; correlation divides by both standard deviations, so it's unitless and between −1 and 1.
  4. Pearson vs Spearman? Pearson: linear relationships, sensitive to outliers. Spearman: Pearson on ranks, measures monotonic relationships, more robust.
  5. Why divide by n − 1 for a sample variance? The sample mean is fitted to the sample, so squared distances to it are too small on average; n − 1 removes that bias (Bessel's correction).
  6. State the central limit theorem and why it matters. Means of many independent values are approximately normal, whatever the original distribution; it justifies normal-based confidence intervals and tests.
  7. What is a p-value? The probability, assuming the null hypothesis, of a result at least as extreme as the one observed. Not the probability the null is true.
  8. What does a 95% confidence interval mean? The procedure captures the true parameter in 95% of repeated samples.
  9. t-test vs chi-square test? t-test: compares means of a numeric variable between two groups. Chi-square: tests whether two categorical variables are independent, using counts.
  10. How would you check if a new model really beats the old one? Compare them on the same data with a paired or permutation test (or cross-validation scores, Episode 26), and check that the gap is larger than what luck produces.

Glossary

Term Meaning Example
Statistics Learning about the truth (a population) from data is the recipe better?
Mean Sum ÷ count 17.9
Median Middle value of the sorted data 6.5
Mode Most common value 6
Outlier A value far from the rest the founder's 120
Range Max − min 53 → 293
Percentile Value below which p% of data lies 90th = 39.5 min
Quartiles (Q1, Q2, Q3) 25th, 50th, 75th percentiles 20.6, 26.8, 32.7
Interquartile range (IQR) Q3 − Q1 12.1
Robust statistic Barely moved by outliers median, IQR
Skewness Lopsidedness of a distribution salaries: 2.11
Right / left skew Long tail of big / small values incomes / easy-test scores
Covariance Average product of deviations from the means −184.08
Pearson correlation r Covariance ÷ (σx σy), between −1 and 1 −0.9977
Linear relationship Points near a straight line temperature vs cups
Rank Position in sorted order 1 … 8
Spearman correlation Pearson on ranks; measures monotonic relationships 1.0
Monotonic Always rising (or always falling) hours vs marks
Correlation vs causation Moving together ≠ one causing the other umbrellas and chai
Confounder Hidden common cause of two variables rain
Population Everyone we care about all adults in a city
Sample The part we measure 5 people
Parameter True population number μ = 165
Statistic The same number from a sample x̄
Bias (of an estimator) Systematic error on average dividing by n: 39.2 vs 49
Bessel's correction, ddof=1 Divide by n − 1 for sample variance 49.4
Central limit theorem Means of many independent values are ≈ normal skew 1.99 → 0.27
Standard error σ / √n: spread of the sample mean 1.41
Confidence interval Range that catches the parameter with a stated frequency 164.05 to 166.57
A/B test Experiment comparing two versions old vs masala chai
Hypothesis test Deciding if data is consistent with a claim is the gap luck?
Null hypothesis H₀ "No effect / no difference" recipe changes nothing
Alternative hypothesis H₁ "There is an effect" masala sells more
p-value P(result at least this extreme | H₀) 0.0234
Significance level α Cut-off for rejecting H₀ 0.05
Permutation (shuffle) test p-value by shuffling group labels 10,000 shuffles
t-test Tests a difference in means using the t statistic ttest_ind
Chi-square test Tests independence of categorical variables p = 0.0009
Expected counts Counts if variables were independent 108 morning teas
Bootstrap Resampling the data with replacement to estimate uncertainty CI for the median
Type I / type II error False positive / false negative decision Episode 24

Summary

  • Mean, median, mode describe the centre; outliers drag the mean, so use the median for skewed data.
  • Percentiles, quartiles and the IQR describe spread robustly; the range is fragile. Skewness measures lopsidedness.
  • Covariance has units; Pearson r removes them; Spearman works on ranks. Correlation is not causation: watch for confounders.
  • A sample estimates a population; divide by n − 1 for sample variance.
  • The central limit theorem makes averages normal; the standard error σ/√n sets the width of a confidence interval.
  • A hypothesis test asks whether luck could explain a result: H₀, p-value, α = 0.05; t-test for means, chi-square test for categories.

Next episode: Maths for ML 10, likelihood and information theory: maximum likelihood, log-likelihood, MAP, entropy, cross-entropy and KL divergence, and where loss functions come from.