SyntaxPilot
Courses/Data Analytics with Python/Lesson 35
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Data Analytics with Python · Lesson 35 of 90

Descriptive statistics 2: spread and percentiles

measures of dispersionrangevariancestandard deviationpopulation vs sample varianceBessel's correction (n-1)quartilespercentilesquantile()interquartile range (IQR)median absolute deviation (MAD)coefficient of variationfive-number summaryreading a box plot

These are the notes for Data Analytics with Python, episode 35. Last episode gave you seven averages. But two stalls can have exactly the same average and completely different days. Today you invent the numbers that measure spread: the range, variance and standard deviation, the difference between population and sample variance (Bessel's correction, ddof), percentiles, quartiles and quantile(), the interquartile range (IQR), the median absolute deviation (MAD), the coefficient of variation, the five-number summary and how to read a box plot. Every term is defined the first time it appears and collected in the Glossary.

Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, SciPy 1.18.1, matplotlib 3.11.2, on a server with no screen). Outputs are copied from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its last line, as in the video.

The problem: same average, different days

import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
a = pd.Series([95, 100, 105, 98, 102, 100, 97, 103, 99, 101])   # Pune
b = pd.Series([20, 180, 60, 140, 100, 40, 160, 100, 80, 120])   # Nashik
print("mean  :", a.mean(), b.mean())
print("median:", a.median(), b.median())
mean  : 100.0 100.0
median: 100.0 100.0

Identical centres. But Pune's bills are all between 95 and 105, and Nashik's run from 20 to 180.

Real-life example: two buses. Bus A arrives at 9:00 every day, give or take a minute. Bus B also arrives at 9:00 on average, but anywhere from 8:40 to 9:20. Same average, very different life. What you need is a second number: how spread out the values are around the centre. That is dispersion (or spread), and a number that measures it is a measure of dispersion.

Range

The range is the largest value minus the smallest.

print("range a:", a.max() - a.min())
print("range b:", b.max() - b.min())
a2 = pd.concat([a, pd.Series([300])], ignore_index=True)
print("range a + one big order:", a2.max() - a2.min())
range a: 10
range b: 160
range a + one big order: 205

The range sees the difference (10 vs 160), but one 300-rupee order makes steady Pune's range 205, bigger than Nashik's. The range uses only two values, the most extreme ones, so one outlier (a value far from the rest) controls it completely.

Variance and standard deviation, step by step

Step 1: deviations. A deviation is a value minus the mean. Try averaging them:

d = b - b.mean()                 # deviation from the mean
print(d.tolist())
print("mean deviation:", d.mean())
[-80.0, 80.0, -40.0, 40.0, 0.0, -60.0, 60.0, 0.0, -20.0, 20.0]
mean deviation: 0.0

Always zero, for every dataset: negative deviations exactly cancel positive ones, because the mean is the balance point of the data.

Step 2: square them, so minus signs disappear. Step 3: the mean of the squared deviations is the variance. Step 4: its square root is the standard deviation.

d = b - b.mean()
var = (d ** 2).mean()            # mean squared deviation
print("variance:", var, "rupees squared")
print("std dev :", round(var ** 0.5, 1), "rupees")
variance: 2400.0 rupees squared
std dev : 49.0 rupees

Variance is in squared units (rupees², which means nothing to a chai seller), so the standard deviation, back in rupees, is what you report: roughly how far a typical value is from the mean. Small standard deviation: values huddle near the mean. Large: they are scattered.

Why squares and not absolute values? Squares have nice mathematical properties (variances of independent things add up, and they lead to the normal distribution's formulas). The absolute-value version exists too: the mean absolute deviation, and the more robust MAD below.

Population vs sample: ddof and Bessel's correction

print("pandas b.std()  :", round(b.std(), 1))
print("numpy np.std(b) :", round(np.std(b.to_numpy()), 1))
print("pandas ddof=0   :", round(b.std(ddof=0), 1))
print("Pune vs Nashik  :", round(a.std(), 1), round(b.std(), 1))
pandas b.std()  : 51.6
numpy np.std(b) : 49.0
pandas ddof=0   : 49.0
Pune vs Nashik  : 2.9 51.6

Same data, two answers, and both are right:

Divide squared deviations by Called Default in
The data is the whole population n population variance / std NumPy np.var, np.std; scipy.stats.zscore
The data is a sample, used to estimate the population n − 1 sample variance / std pandas .var(), .std(), describe()

ddof means delta degrees of freedom: the functions divide by n − ddof. pandas uses ddof=1, NumPy ddof=0. Pass it explicitly when it matters.

Why n − 1? Proof by simulation:

import sys; sys.tracebacklimit = 0
import numpy as np
rng = np.random.default_rng(35)
pop = rng.normal(100, 50, 100_000)          # every bill ever: the population
samples = rng.choice(pop, size=(20_000, 5))  # 20,000 samples of 5 bills
print("true variance   :", round(pop.var()))
print("divide by n     :", round(samples.var(axis=1, ddof=0).mean()))
print("divide by n - 1 :", round(samples.var(axis=1, ddof=1).mean()))
true variance   : 2484
divide by n     : 1988
divide by n - 1 : 2486

Dividing by n underestimates the true variance (by a factor of (n − 1)/n = 4/5 for samples of 5). A sample's own mean sits in the middle of its values, so distances to it are smaller than distances to the true population mean. Dividing by n − 1 corrects for that: Bessel's correction. On average the n − 1 version hits the target (it is unbiased). With large samples, n and n − 1 are nearly equal.

(axis=1 means "along each row", so each of the 20,000 rows, one sample, gets its own variance.)

Percentiles, quartiles and quantile()

Real-life example: exam rank. "You scored better than 90% of students" means you are at the 90th percentile. A percentile is the value below which that percentage of the data falls. Cut the sorted data into four equal groups and the three cut points are the quartiles: Q1 = 25th percentile, Q2 = 50th = the median, Q3 = 75th percentile. In code the percentage is written as a fraction between 0 and 1 and called a quantile: the 0.9 quantile is the 90th percentile.

print(b.sort_values().tolist())
print(b.quantile([0.25, 0.5, 0.75]))
print("90th percentile:", b.quantile(0.9))
[20, 40, 60, 80, 100, 100, 120, 140, 160, 180]
0.25     65.0
0.50    100.0
0.75    135.0
dtype: float64
90th percentile: 162.0

65 is not in the data: with 10 values, the 25th percentile's position is 0.25 × (10 − 1) = 2.25, between the 3rd value (60) and the 4th (80), so pandas interpolates a quarter of the way: 65. That is the default interpolation="linear"; other methods ("lower", "nearest", "midpoint") exist and give slightly different answers on small data.

Delivery apps state promises as percentiles ("90% of orders within 30 minutes") because a percentile tells customers what they can rely on, which an average cannot.

print(b.quantile(25))
ValueError: percentiles should all be in the interval [0, 1]

pandas quantile wants a fraction (0.25). NumPy's np.percentile(b, 25) is the one that takes 0–100.

Robust spread: IQR and MAD

  • Interquartile range (IQR) = Q3 − Q1: the width of the middle 50% of the data.
  • Median absolute deviation (MAD) = the median of |value − median|.

Both ignore how extreme the extremes are, just like the median.

from scipy import stats
b2 = pd.concat([b, pd.Series([5000])], ignore_index=True)
for s in (b, b2):
    q1, q3 = s.quantile(0.25), s.quantile(0.75)
    print("std", round(s.std(), 1), "| IQR", q3 - q1,
          "| MAD", stats.median_abs_deviation(s))
std 51.6 | IQR 70.0 | MAD 40.0
std 1478.2 | IQR 80.0 | MAD 40.0

One 5,000-rupee order multiplies the standard deviation by almost 30, because squaring makes big deviations enormous. The IQR moves a little and the MAD not at all: they are robust measures of spread. Report them (with the median) when data has outliers or is skewed. (scipy.stats.median_abs_deviation has a scale="normal" option that multiplies by about 1.4826 so it estimates the standard deviation for bell-shaped data.)

Coefficient of variation

chai = pd.Series([20, 180, 60, 140, 100, 40, 160, 100, 80, 120])
catering = pd.Series([4500, 5600, 4800, 5200, 5000, 4400, 5500, 5000])
for name, s in [("chai", chai), ("catering", catering)]:
    cv = s.std() / s.mean()
    print(f"{name:9} mean {s.mean():6.0f}  std {s.std():5.0f}  CV {cv:.0%}")
chai      mean    100  std    52  CV 52%
catering  mean   5000  std   431  CV 9%

Catering's standard deviation (431) is bigger, but relative to its size it varies far less. The coefficient of variation (CV) = std ÷ mean, the spread as a share of the mean. Use it to compare spread between things on different scales (prices of phones vs chai, weights of elephants vs mice). It only makes sense for positive data measured from a true zero (not for temperatures in °C).

f"{cv:.0%}" formats a fraction as a percentage with no decimals.

The five-number summary and describe()

import matplotlib.pyplot as plt
rng = np.random.default_rng(35)
bills = pd.Series(np.r_[rng.normal(110, 25, 196).round(), [260, 275, 300, 15]])
print(bills.describe().round(1))
count    200.0
mean     114.6
std       33.8
min       15.0
25%       94.8
50%      111.5
75%      131.2
max      300.0
dtype: float64

std in describe() is the sample standard deviation (n − 1). The last five rows, minimum, Q1, median, Q3, maximum, are the five-number summary: centre, spread and extremes in five numbers. (np.r_[...] joins arrays and lists into one array.)

Reading a box plot

q1, med, q3 = bills.quantile([0.25, 0.5, 0.75])
iqr = q3 - q1
print("Q1", q1, "median", med, "Q3", q3, "IQR", iqr)
print("fences:", q1 - 1.5 * iqr, q3 + 1.5 * iqr)
fig, ax = plt.subplots(figsize=(10, 4.2))
ax.boxplot(bills, orientation="horizontal", widths=0.5)
for x, t, y in [(q1, "Q1", 1.36), (med, "median", 1.47), (q3, "Q3", 1.36)]:
    ax.annotate(t, (x, 1.26), (x, y), ha="center", fontsize=13,
                arrowprops={"arrowstyle": "->"})
ax.set(xlabel="bill (rupees)", yticks=[], ylim=(0.6, 1.56))
fig.tight_layout(); fig.savefig("box.png")
Q1 94.75 median 111.5 Q3 131.25 IQR 36.5
fences: 40.0 186.0

box.png

Part What it shows
Left edge of the box Q1 (25th percentile)
Line inside the box the median
Right edge of the box Q3 (75th percentile)
Box width the IQR: the middle 50% of values
Fences (not drawn) Q1 − 1.5 × IQR and Q3 + 1.5 × IQR (here 40 and 186)
Whiskers reach the furthest data point still inside the fences (here 40 and 185)
Dots points beyond the fences: possible outliers (15, 192, 260, 275, 300)

This is the IQR outlier rule from episode 23, drawn as a picture. A long whisker or box half on one side signals skew. (ax.annotate(text, xy, xytext, arrowprops=...) writes text at xytext with an arrow pointing to xy; in matplotlib 3.11 orientation="horizontal" replaces the older vert=False.)

Common mistakes

Code / habit Real error / symptom Why Fix
b.quantile(25) ValueError: percentiles should all be in the interval [0, 1] quantile wants a fraction b.quantile(0.25) or np.percentile(b, 25)
b.std() vs np.std(b) no error; 51.6 vs 49.0 pandas ddof=1, NumPy ddof=0 pass ddof= explicitly
(b - b.mean()).mean() as spread no error; always 0 deviations cancel square them: variance, std
std of data with outliers no error; 51.6 → 1478.2 squaring magnifies big values IQR or MAD
reporting variance to a manager no error; "2400 rupees squared" squared units report the standard deviation
comparing std across different scales no error; misleading bigger values have bigger std coefficient of variation
range as the only spread measure no error; one outlier decides it uses only min and max IQR / std

Mini project: delivery reliability report

Three stalls deliver in about 12 minutes on average. Which one can promise a delivery time?

import numpy as np, pandas as pd
from scipy import stats
rng = np.random.default_rng(11)
df = pd.DataFrame({"stall": np.repeat(["Pune", "Nashik", "Mumbai"], 100),
                   "minutes": np.r_[rng.normal(12, 2, 100), rng.normal(12, 6, 100),
                                    rng.gamma(2, 6, 100)].round(1)})
def iqr(s): return s.quantile(0.75) - s.quantile(0.25)
def p90(s): return s.quantile(0.9)
def mad(s): return stats.median_abs_deviation(s)
def cv(s): return s.std() / s.mean()
report = df.groupby("stall")["minutes"].agg(["mean", "median", "std", iqr, mad, p90, cv]).round(2)
print(report)
         mean  median   std   iqr   mad    p90    cv
stall                                               
Mumbai  12.15   12.35  6.89  8.57  4.65  20.50  0.57
Nashik  12.05   12.10  5.59  7.33  3.50  19.27  0.46
Pune    12.05   12.10  1.84  2.48  1.25  14.41  0.15

All three average about 12 minutes, but Pune can honestly promise "90% within 15 minutes"; Mumbai would need to promise 21. Named functions passed to agg become column names (iqr, mad, p90, cv). (np.repeat repeats each label 100 times.)

Practice

1. Your daily screen time (hours) for a week: 3.5, 4.0, 2.5, 6.0, 5.0, 3.0, 4.5. Compute the standard deviation both ways. Which one estimates your usual variability?

Solution
import numpy as np, pandas as pd
screen = pd.Series([3.5, 4.0, 2.5, 6.0, 5.0, 3.0, 4.5])   # hours per day
print("pandas std (ddof=1):", round(screen.std(), 3))
print("numpy std  (ddof=0):", round(np.std(screen.to_numpy()), 3))
print("pandas ddof=0      :", round(screen.std(ddof=0), 3))
pandas std (ddof=1): 1.205
numpy std  (ddof=0): 1.116
pandas ddof=0      : 1.116

One week is a sample of your life, so the ddof=1 value (1.205 hours) is the better estimate.

2. Commute times (minutes): 32, 35, 31, 40, 33, 34, 58, 36, 33, 37. Give the median, the 90th percentile, the IQR and the box-plot fences. Is 58 an outlier?

Solution
import pandas as pd
commute = pd.Series([32, 35, 31, 40, 33, 34, 58, 36, 33, 37])   # minutes
print("median:", commute.median())
print("90th percentile:", commute.quantile(0.9))
q1, q3 = commute.quantile([0.25, 0.75])
print("IQR:", q3 - q1, "fences:", q1 - 1.5 * (q3 - q1), q3 + 1.5 * (q3 - q1))
median: 34.5
90th percentile: 41.8
IQR: 3.75 fences: 27.375 42.375

Yes: 58 is above the upper fence (42.375).

3. Phone prices vs chai prices: which is relatively more variable?

Solution
import pandas as pd
phone = pd.Series([15000, 18000, 22000, 16000, 19000])
chai = pd.Series([10, 15, 12, 20, 8])
for name, s in [("phone", phone), ("chai", chai)]:
    print(name, round(s.std(), 1), f"CV {s.std() / s.mean():.0%}")
phone 2738.6 CV 15%
chai 4.7 CV 36%

Chai: its standard deviation is tiny in rupees but 36% of its mean.

4. Compute the sample variance of 4, 8, 6, 5, 3, 7, 9, 5 by hand in code and check against pandas.

Solution
import pandas as pd
s = pd.Series([4, 8, 6, 5, 3, 7, 9, 5])
d = s - s.mean()
print("mean:", s.mean(), "sum of deviations:", d.sum())
print("variance (n-1):", (d ** 2).sum() / (len(s) - 1), "| pandas:", s.var())
print("std:", round(s.std(), 3))
mean: 5.875 sum of deviations: 0.0
variance (n-1): 4.125 | pandas: 4.125
std: 2.031

Quick quiz

x = pd.Series([50, 50, 50, 50])
y = pd.Series([10, 30, 70, 90])
print(x.mean(), y.mean())  # 50.0 50.0

Which has the larger standard deviation? A. x B. y C. Equal, same mean D. Can't tell without a chart

Answer

B. y. All of x's deviations are 0, so its std is 0. y's deviations are ±20 and ±40: sample variance = (1600 + 400 + 400 + 1600) / 3 ≈ 1333.3, std ≈ 36.5.

Interview questions

  1. Why not use the mean of the deviations as spread? It is always zero; positive and negative deviations cancel.
  2. Variance vs standard deviation? Variance is the mean squared deviation (squared units); std is its square root (original units).
  3. Why divide by n − 1 for a sample? The sample mean is fitted to the sample, which makes deviations too small; n − 1 (Bessel's correction) makes the variance estimate unbiased.
  4. What is ddof? Delta degrees of freedom: divide by n − ddof. pandas defaults to 1, NumPy to 0.
  5. Percentile vs quantile? Same idea; percentiles use 0–100, quantiles use 0–1.
  6. What is the IQR and why use it? Q3 − Q1, the spread of the middle 50%; robust to outliers.
  7. What is MAD? Median of absolute deviations from the median; a very robust spread measure.
  8. When is the coefficient of variation useful? Comparing variability across variables with different scales or units.
  9. What are the five numbers in the five-number summary? Min, Q1, median, Q3, max.
  10. How are box plot whiskers and outlier dots defined (default)? Whiskers reach the furthest point within 1.5 × IQR of the box; points beyond are drawn individually.
  11. Why do companies quote the 90th or 99th percentile of latency or delivery time? Averages hide the slow cases that customers actually notice.

Glossary

Term Meaning Example
Spread / dispersion How far values sit from the centre Pune low, Nashik high
Measure of dispersion A number that measures spread range, std, IQR
Range Max − min Nashik: 160
Outlier A value far from the rest the 300-rupee order
Deviation Value − mean 20 − 100 = −80
Variance Mean of squared deviations (units²) 2400 rupees²
Standard deviation (std) √variance, in the data's units 49 rupees
Population variance Divide by n np.var(x)
Sample variance Divide by n − 1 s.var()
Bessel's correction Using n − 1 for a sample to avoid underestimating 1988 → 2486
Unbiased Correct on average over many samples ddof=1 variance
ddof Delta degrees of freedom: divide by n − ddof s.std(ddof=0)
Percentile Value below which p% of the data falls 90th percentile = 162
Quantile Percentile written as a fraction 0–1 quantile(0.9)
Quartiles (Q1, Q2, Q3) 25th, 50th, 75th percentiles 65, 100, 135
Interpolation Estimating a value between two data points Q1 = 65 between 60 and 80
Interquartile range (IQR) Q3 − Q1 70
Median absolute deviation (MAD) Median of |x − median| 40
Robust Barely affected by outliers IQR, MAD, median
Coefficient of variation (CV) std ÷ mean chai 52%, catering 9%
Five-number summary Min, Q1, median, Q3, max describe() rows
Box plot Chart of the five-number summary plus outliers ax.boxplot(...)
Whisker Line from the box to the furthest point inside the fences 40 and 185
Fence Q1 − 1.5·IQR and Q3 + 1.5·IQR 40 and 186

Summary

  • The same average can hide very different data; you need a measure of dispersion.
  • Range uses only two values. Variance averages squared deviations; the standard deviation brings it back to real units.
  • Population: divide by n; sample: by n − 1 (Bessel's correction). pandas ddof=1, NumPy ddof=0.
  • Percentiles/quantiles and quartiles describe positions; pandas quantile takes 0–1.
  • IQR and MAD are robust spread; the CV compares spread across scales.
  • A box plot draws the five-number summary and flags points beyond 1.5 × IQR.
  • Next episode: the shape of data: skewness, kurtosis, z-scores and transformations.