These are the notes for Data Analytics with Python, episode 45. Episode 44 showed that about 95% of sample means land within 1.96 standard errors of the truth. Here we turn that around: from one sample, build a range that catches the true value 95% of the time. You'll meet the point estimate and interval estimate, the confidence interval (CI), confidence level, margin of error and critical value, the t-distribution and degrees of freedom, intervals for a mean and a proportion (including the Wilson interval), how to interpret a CI correctly, the bootstrap (resampling), and the sample size needed for a target margin. Every term is defined the first time it appears and collected in the Glossary.
Every program below was run for real (Python 3.13, NumPy 2.5.3, SciPy 1.18.1, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Every program starts with these lines, which build 100,000 delivery times for TiffinBox, the made-up tiffin delivery app:
import sys; sys.tracebacklimit = 0
import numpy as np
from scipy import stats
rng = np.random.default_rng(45)
times = rng.gamma(9, 4, 100_000).round(1) # every delivery, minutes (made up)
The times follow a gamma distribution (a positive, slightly right-skewed shape) with a true mean of about 36 minutes. In real life you would only have the sample; here we know the truth so we can check every method.
The problem: one number hides the wobble
week1 = rng.choice(times, 40) # 40 timed deliveries
week2 = rng.choice(times, 40)
print("week 1 average:", round(week1.mean(), 1), "min")
print("week 2 average:", round(week2.mean(), 1), "min")
print("truth (unknown in real life):", round(times.mean(), 1), "min")
week 1 average: 34.7 min
week 2 average: 36.0 min
truth (unknown in real life): 36.0 min
The boss printed "average delivery: 34.7 minutes" in an ad; week two disagreed.
- Point estimate: a single number from a sample used as the best guess for a parameter (34.7 min). It says nothing about how wrong it might be.
- Interval estimate: a range of plausible values (31.2 to 38.2 min) with a stated chance that the method catches the truth.
Real-life example: spear vs net
| Fishing | Estimation |
|---|---|
| a spear hits one exact spot and almost always misses | point estimate |
| a net covers a range and usually catches the fish | interval estimate |
| a wider net catches more often but says less about where the fish is | confidence level: 95% vs 99% |
Building a confidence interval
- Confidence interval (CI) = point estimate ± margin of error.
- Margin of error (MOE) = critical value × standard error.
- Critical value: how many standard errors to go either side. For a 95% interval from the normal curve it's
stats.norm.ppf(0.975)= 1.96 (2.5% is left in each tail). - Confidence level: the share of intervals built this way that catch the truth (95%, 90%, 99%).
x = rng.choice(times, 40)
se = x.std(ddof=1) / np.sqrt(len(x))
z = stats.norm.ppf(0.975) # critical value for 95%
moe = z * se # margin of error
print("mean", round(x.mean(), 1), " SE", round(se, 2), " z", round(z, 2))
print(f"95% CI: {x.mean() - moe:.1f} to {x.mean() + moe:.1f} minutes")
mean 34.7 SE 1.78 z 1.96
95% CI: 31.2 to 38.2 minutes
Margin of error ≈ 1.96 × 1.78 ≈ 3.5 minutes; the interval 31.2-38.2 contains the true 36.0.
Small samples miss too often
Does the 95% promise hold? Build 10,000 intervals and count how many catch the true mean:
def caught(n, crit, reps=10_000):
hits = 0
for _ in range(reps):
x = rng.choice(times, n)
moe = crit(n) * x.std(ddof=1) / np.sqrt(n)
hits += abs(x.mean() - times.mean()) < moe
return hits / reps
z = lambda n: 1.96
print("n = 40, z 1.96:", caught(40, z))
print("n = 8, z 1.96:", caught(8, z))
n = 40, z 1.96: 0.9426
n = 8, z 1.96: 0.9067
With 8 deliveries, our "95%" interval catches the truth only 90.7% of the time. Why: we plugged in the sample's s as if it were the true σ. With 8 values, s is itself a shaky guess, often too small, so the net comes out too narrow.
The t-distribution and degrees of freedom
- t-distribution (Student's t): a bell-shaped distribution like the standard normal but with fatter tails, which account for the extra uncertainty of estimating σ with s.
- Degrees of freedom (df): the number of values free to vary once the estimate is fixed; for one mean, df = n − 1. The fewer the df, the fatter the tails. As df grows, t approaches the normal.
for df in [3, 7, 29, 100]:
print(f"df = {df:>3}: t critical = {stats.t.ppf(0.975, df):.2f}")
def caught(n, reps=10_000):
hits = 0
for _ in range(reps):
x = rng.choice(times, n)
moe = stats.t.ppf(0.975, n - 1) * x.std(ddof=1) / np.sqrt(n)
hits += abs(x.mean() - times.mean()) < moe
return hits / reps
print("n = 8, t critical:", caught(8))
df = 3: t critical = 3.18
df = 7: t critical = 2.36
df = 29: t critical = 2.05
df = 100: t critical = 1.98
n = 8, t critical: 0.9412
With t (2.36 for 7 df) the 8-delivery intervals catch the truth 94.1% of the time, back near 95%. The small remaining gap comes from the skew in delivery times (t assumes roughly normal data; the CLT helps more as n grows). Rule: for a mean, use t. For large samples it's practically the same as z.
CI for a mean: x̄ ± t(0.975, n − 1) × s / √n. In SciPy, one line:
x = rng.choice(times, 8) # the small kitchen: 8 deliveries
lo, hi = stats.t.interval(0.95, df=len(x) - 1, loc=x.mean(), scale=stats.sem(x))
print("8 deliveries:", x)
print(f"mean {x.mean():.1f}, 95% CI {lo:.1f} to {hi:.1f} minutes")
lo, hi = stats.t.interval(0.99, df=len(x) - 1, loc=x.mean(), scale=stats.sem(x))
print(f"99% CI {lo:.1f} to {hi:.1f} (more confident = wider)")
8 deliveries: [43. 21. 46.8 38.4 68.1 19.2 44.6 44. ]
mean 40.6, 95% CI 27.7 to 53.6 minutes
99% CI 21.5 to 59.8 (more confident = wider)
Eight deliveries don't tell you much, and the wide interval says so honestly. Higher confidence always costs a wider interval.
What 95% really means
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(13, 4.8)); miss = 0
for i in range(100):
x = rng.choice(times, 40)
lo, hi = stats.t.interval(0.95, 39, loc=x.mean(), scale=stats.sem(x))
ok = lo < times.mean() < hi; miss += not ok
ax.plot([i, i], [lo, hi], color="#2b6cb0" if ok else "#e4572e", lw=2.4)
ax.plot(i, x.mean(), "o", color="#1a202c", ms=3)
ax.axhline(times.mean(), color="#1a202c", ls="--", lw=1.5, label="true mean")
ax.set_title(f"100 samples of 40 deliveries, 100 95% intervals: {100 - miss} catch the true mean, {miss} miss (red)")
ax.set_xlabel("sample number"); ax.set_ylabel("minutes"); ax.legend(loc="upper right")
fig.tight_layout(); fig.savefig("hundred.png", dpi=110)
print("missed:", miss)
missed: 7

Interpreting a CI correctly: - ✅ "If we repeated this sampling many times, about 95% of the intervals built this way would contain the true mean. We are 95% confident in this one." - ❌ "There is a 95% probability that the true mean is between 31.2 and 38.2." The true mean is a fixed number; this particular interval either contains it or doesn't. The 95% describes the method. - ❌ "95% of deliveries take between 31.2 and 38.2 minutes." A CI is about the mean, not individual values (individual deliveries vary far more).
CI for a proportion
CI for a proportion: p̂ ± 1.96 × √(p̂(1 − p̂)/n), where p̂ ("p-hat") is the sample share.
late, n = 23, 400 # 23 of 400 deliveries were late
p = late / n
moe = 1.96 * np.sqrt(p * (1 - p) / n)
print(f"late share {p:.3f}, 95% CI {p - moe:.3f} to {p + moe:.3f}")
k, m = 0, 20 # new kitchen: 0 complaints in 20
q = k / m
print("0 of 20, same formula:", q - 1.96 * np.sqrt(q * (1 - q) / m),
"to", q + 1.96 * np.sqrt(q * (1 - q) / m))
ci = stats.binomtest(k, m).proportion_ci(method="wilson")
print(f"0 of 20, Wilson: {ci.low:.3f} to {ci.high:.3f}")
late share 0.058, 95% CI 0.035 to 0.080
0 of 20, same formula: 0.0 to 0.0
0 of 20, Wilson: 0.000 to 0.161
The simple formula (the Wald interval) works for 23 of 400, but for 0 of 20 it gives "0 to 0": certainty from 20 deliveries, which is nonsense. It breaks when p̂ is near 0 or 1 or n is small (a common check: at least 10 successes and 10 failures). The Wilson interval (stats.binomtest(k, n).proportion_ci(method="wilson")) stays sensible: up to 16.1% complaints is still plausible.
The bootstrap
The median delivery time has no simple standard-error formula. The bootstrap gets one from the data itself.
- Resampling: drawing new samples from your own sample.
- Bootstrap: treat the sample as a mini population; draw a bootstrap sample of the same size with replacement (some values repeat, some are left out), compute the statistic, repeat thousands of times. The spread of those statistics estimates the sampling distribution.
- Percentile bootstrap interval: the 2.5th to 97.5th percentile of the bootstrap statistics.
A real error first:
x = rng.choice(times, 30)
res = stats.bootstrap(x, np.median)
numpy.exceptions.AxisError: `axis` is out of bounds for array of dimension 0
stats.bootstrap expects a sequence of samples, so it read each number as a separate sample. Wrap the data in a tuple: (x,).
x = rng.choice(times, 30) # our ONE sample of 30
meds = [np.median(rng.choice(x, len(x), replace=True)) for _ in range(10_000)]
print("sample median:", np.median(x))
print("bootstrap 95% CI:", np.percentile(meds, [2.5, 97.5]).round(1))
res = stats.bootstrap((x,), np.median, confidence_level=0.95, rng=rng)
ci = res.confidence_interval
print(f"scipy bootstrap : {ci.low:.1f} to {ci.high:.1f}")
print("true median :", np.median(times))
sample median: 35.25
bootstrap 95% CI: [28. 39.2]
scipy bootstrap : 28.0 to 39.9
true median : 34.7
The hand-made percentile interval (28 to 39.2) and SciPy's (28 to 39.9; SciPy uses the more refined BCa method by default) both contain the true median 34.7. The bootstrap works for medians, percentiles, ratios, correlations, almost anything. Limit: it only knows what's in your sample; a tiny or biased sample still gives a misleading interval.
Sample size for a target margin
Before collecting data, choose the margin of error E you want and solve for n:
- Mean: n = (z × σ / E)² (σ from a pilot sample or past data).
- Proportion: n = z² × p(1 − p) / E²; with no idea of p, use p = 0.5, the worst case.
sd, E = 12, 2 # guessed sd, wanted margin
print("deliveries needed:", int(np.ceil((1.96 * sd / E) ** 2)))
p, E = 0.5, 0.03 # a poll: +/- 3 percentage points
print("people needed for a poll:", int(np.ceil(1.96**2 * p * (1 - p) / E**2)))
deliveries needed: 139
people needed for a poll: 1068
This is why many opinion polls interview about a thousand people: ±3 points at 95% confidence. (Always round up.)
Common mistakes
| Code / habit | Real error / symptom | Why | Fix |
|---|---|---|---|
stats.bootstrap(x, np.median) |
numpy.exceptions.AxisError:axisis out of bounds for array of dimension 0 |
expects a sequence of samples | stats.bootstrap((x,), np.median) |
| 1.96 with 8 values | no error; catches the truth only 90.7% of the time | s is a shaky guess for σ | t critical with df = n − 1 |
| simple proportion formula for 0 of 20 | no error; "0 to 0" | fails near 0 or 1 and for small n | Wilson interval |
| "95% chance the truth is in 31.2-38.2" | no error; wrong meaning | 95% describes the method | "95% confident" |
| reading a CI as the range of individual values | no error; far too narrow | a CI is about the mean | use the data's spread for individuals |
x.std() in NumPy for s |
no error; slightly too narrow | default ddof=0 |
x.std(ddof=1) or stats.sem(x) |
| rounding the sample size down | no error; margin slightly too big | formula gives a minimum | np.ceil |
Mini project: an honest ad line
Write the ad from 40 deliveries with its interval, then find how many deliveries to time for a margin of ±1 minute, and check it.
x = rng.choice(times, 40) # this week's 40 deliveries
lo, hi = stats.t.interval(0.95, len(x) - 1, loc=x.mean(), scale=stats.sem(x))
print(f"ad line: average delivery {x.mean():.0f} min (95% CI {lo:.0f}-{hi:.0f} min)")
E = 1 # we want +/- 1 minute next time
n = int(np.ceil((1.96 * x.std(ddof=1) / E) ** 2))
print("deliveries to time for +/- 1 minute:", n)
y = rng.choice(times, n)
lo, hi = stats.t.interval(0.95, n - 1, loc=y.mean(), scale=stats.sem(y))
print(f"with {n}: {y.mean():.1f} min, 95% CI {lo:.1f} to {hi:.1f}")
ad line: average delivery 35 min (95% CI 31-38 min)
deliveries to time for +/- 1 minute: 485
with 485: 36.4 min, 95% CI 35.3 to 37.5
Practice
1. A 90% t-interval for these 10 values: 12, 15, 9, 14, 11, 13, 16, 10, 12, 14.
Solution
import numpy as np
from scipy import stats
x = np.array([12, 15, 9, 14, 11, 13, 16, 10, 12, 14])
lo, hi = stats.t.interval(0.90, df=len(x) - 1, loc=x.mean(), scale=stats.sem(x))
print(round(x.mean(), 1), round(lo, 2), round(hi, 2))
12.6 11.31 13.89
2. A 95% Wilson interval for 3 defects in 50 items.
Solution
from scipy import stats
ci = stats.binomtest(3, 50).proportion_ci(confidence_level=0.95, method="wilson")
print(round(3 / 50, 3), round(ci.low, 3), round(ci.high, 3))
0.06 0.021 0.162
3. A bootstrap 95% interval for the 90th percentile of 60 delivery times.
Solution
import numpy as np
from scipy import stats
rng = np.random.default_rng(9)
x = rng.gamma(9, 4, 60)
res = stats.bootstrap((x,), lambda s, axis: np.percentile(s, 90, axis=axis), rng=rng)
print(round(np.percentile(x, 90), 1), round(res.confidence_interval.low, 1), round(res.confidence_interval.high, 1))
47.0 42.1 54.8
The statistic function must accept an axis argument, because SciPy computes many resamples at once.
Quick quiz
# 95% CI from n = 100: 50 ± 4
# Which change makes the interval NARROWER?
A. 99% confidence instead of 95% B. n = 400 instead of 100 C. Use z instead of t to look precise D. Drop the largest values
Answer
B. Margin = critical value × s/√n; 4× the data halves it (± 2). Higher confidence widens it, z with a small n just hides uncertainty, and dropping data is cheating.
Interview questions
- What is a 95% confidence interval? A range from a procedure that catches the true parameter in 95% of repeated samples.
- Why use t instead of z for a mean? σ is unknown and estimated by s; t's fatter tails account for that, especially for small n.
- What are degrees of freedom for a one-sample t-interval? n − 1.
- How does the width change with n and confidence level? Narrower with larger n (∝ 1/√n), wider with higher confidence.
- When does the simple proportion interval fail? Small n or p̂ near 0 or 1; use Wilson.
- Explain the bootstrap. Resample the sample with replacement many times, recompute the statistic, and use the spread (e.g. percentiles) of the results.
- How many people for ±3 points in a poll? About 1,068 at 95%, using p = 0.5.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Point estimate | One number guessing a parameter | mean 34.7 min |
| Interval estimate | A range of plausible values | 31.2 to 38.2 min |
| Confidence interval (CI) | Estimate ± margin of error, from a method with a known catch rate | 95% CI 31.2-38.2 |
| Confidence level | Share of such intervals that catch the truth | 95% |
| Margin of error (MOE) | Critical value × standard error | 1.96 × 1.78 ≈ 3.5 |
| Critical value | Number of SEs to go either side | 1.96 (z), 2.36 (t, df 7) |
| Standard error (SE) | Spread of an estimate, s/√n | 1.78 |
| t-distribution | Bell with fatter tails, for estimated σ | stats.t |
| Degrees of freedom (df) | Values free to vary; n − 1 for one mean | 7 for n = 8 |
stats.t.interval |
SciPy's t confidence interval | t.interval(0.95, df, loc, scale) |
| CI for a mean | x̄ ± t × s/√n | 27.7-53.6 for 8 deliveries |
| CI for a proportion | p̂ ± 1.96 √(p̂(1 − p̂)/n) | 3.5%-8.0% late |
| p̂ (p-hat) | Sample proportion | 23/400 = 0.058 |
| Wald interval | The simple proportion formula above | fails for 0 of 20 |
| Wilson interval | A proportion interval that works near 0 or 1 | 0-16.1% |
| Coverage | How often intervals actually catch the truth | 90.7% with z, n = 8 |
| Resampling | Drawing new samples from your sample | rng.choice(x, len(x), replace=True) |
| Bootstrap | Resample with replacement to estimate a sampling distribution | median CI 28-39.2 |
| Percentile interval | 2.5th to 97.5th percentile of bootstrap statistics | np.percentile(meds, [2.5, 97.5]) |
| BCa | Bias-corrected and accelerated bootstrap, SciPy's default | 28.0-39.9 |
| Sample size formula | n = (z σ / E)² or z² p(1 − p)/E² | 139 deliveries, 1,068 people |
Summary
- A point estimate is one number; an interval estimate shows its uncertainty.
- CI = estimate ± critical value × SE; that product is the margin of error.
- For means use the t-distribution with n − 1 degrees of freedom: z intervals are too narrow for small samples.
- 95% describes how often the method works, not the chance for one interval.
- For proportions use Wilson when n is small or p̂ is near 0 or 1.
- The bootstrap resamples your sample to get intervals for medians, percentiles and more.
- Plan the sample size for the margin you need: about 1,068 people for ±3 points.
- Next episode: hypothesis testing and p-values: is the new kitchen really faster, or was it luck?