SyntaxPilot
Courses/Data Analytics with Python/Lesson 42
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Data Analytics with Python · Lesson 42 of 90

Continuous distributions and the normal curve

probability density function (PDF)cumulative distribution function (CDF)continuous uniformnormal (Gaussian) distributionstandard normalempirical rule (68-95-99.7)exponentiallog-normalQ-Q plotnormality test (Shapiro-Wilk)ppf / inverse CDF

These are the notes for Data Analytics with Python, episode 42. Episode 41's random variables could be counted. Today's can take any value in a range: 4.73 minutes, 151.2 ml. That breaks the probability table, and fixing it gives you the probability density function (PDF) and the cumulative distribution function (CDF) for continuous data, the continuous uniform, the normal (Gaussian) distribution, the standard normal, the empirical rule (68-95-99.7), the inverse CDF (ppf), the exponential and log-normal distributions, and how to check normality with a Q-Q plot and the Shapiro-Wilk test. Every term is defined the first time it appears and collected in the Glossary.

Every program below was run for real (Python 3.13, NumPy 2.5.3, SciPy 1.18.1, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Every program starts with these lines:

import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
from scipy import stats
rng = np.random.default_rng(42)

Why P(exactly 3 minutes) = 0

An auto rickshaw arrives at any moment in the next 10 minutes, every moment equally likely.

wait = rng.uniform(0, 10, 1_000_000)        # auto arrives any time in 10 min
print(wait[:4])
print("P(wait == 3)       =", np.mean(wait == 3))
print("P(2.5 < wait < 3.5) =", round(np.mean((wait > 2.5) & (wait < 3.5)), 3))
[7.73956049 4.3887844  8.5859792  6.97368029]
P(wait == 3)       = 0.0
P(2.5 < wait < 3.5) = 0.1

Not one of a million waits is exactly 3.000 minutes, but 10% fall between 2.5 and 3.5. A wait can be 3.1, 3.01, 3.001, …: infinitely many values share a total probability of 1, so each exact value has probability 0. For a continuous random variable, probability lives on ranges.

PDF and CDF

Real-life example: rain on a terrace. No rain hits one exact mathematical point (a point has no area), but you can say how heavily it rains per square metre (a density), and the rain on a tile is density × area.

  • The probability density function (PDF) f(x) says how much probability there is per unit of x near x. Its height is a density, not a probability, and can be bigger than 1 for narrow distributions.
  • P(a < X < b) = area under the PDF from a to b. The total area is 1.
  • The cumulative distribution function (CDF) F(x) = P(X ≤ x) = area to the left of x. So P(a < X < b) = F(b) − F(a). (For continuous X, < and ≤ give the same answer.)

Continuous uniform: loc and scale

auto = stats.uniform(loc=0, scale=10)        # 0 to 10 minutes
print("density pdf(3)   =", auto.pdf(3))
print("P(wait < 3)      =", auto.cdf(3))
print("P(3 < wait < 7)  =", round(auto.cdf(7) - auto.cdf(3), 3))
print("pdf(12)          =", auto.pdf(12))
wrong = stats.uniform(5, 10)                 # meant: 5 to 10 minutes
print("wrong range:", wrong.ppf(0), "to", wrong.ppf(1))
density pdf(3)   = 0.1
P(wait < 3)      = 0.3
P(3 < wait < 7)  = 0.4
pdf(12)          = 0.0
wrong range: 5.0 to 15.0

stats.uniform(loc, scale) covers loc to loc + scale. Density 0.1 per minute on 0-10 (area 10 × 0.1 = 1); P(wait < 3) = 0.3; P(3 < wait < 7) = 0.4; outside the range the density is 0. Trap: stats.uniform(5, 10) is 5 to 15, not 5 to 10, with no error. Name the arguments: stats.uniform(loc=5, scale=5).

Four continuous shapes

import matplotlib.pyplot as plt
fig, axs = plt.subplots(1, 4, figsize=(16, 4))
x = np.linspace(-2, 12, 500); axs[0].plot(x, stats.uniform(0, 10).pdf(x), lw=3)
axs[0].set_title("uniform(0 to 10): auto wait")
x = np.linspace(130, 170, 500); axs[1].plot(x, stats.norm(150, 5).pdf(x), lw=3)
axs[1].fill_between(x, stats.norm(150, 5).pdf(x), where=x < 140, color="#e4572e")
axs[1].set_title("normal(150, 5): cup ml")
x = np.linspace(0, 30, 500); axs[2].plot(x, stats.expon(scale=5).pdf(x), lw=3)
axs[2].set_title("exponential(mean 5): gap, min")
x = np.linspace(1, 1200, 500); axs[3].plot(x, stats.lognorm(0.6, scale=np.exp(5.5)).pdf(x), lw=3)
axs[3].set_title("log-normal: order value")
fig.tight_layout(); fig.savefig("shapes.png")

shapes.png

The normal distribution

A tea machine pours cups of about 150 ml with standard deviation 5 ml. Quantities made of many small, independent random wobbles added together tend to follow a symmetric bell shape: the normal distribution, also called the Gaussian. It is set by two numbers: the mean μ (loc) and the standard deviation σ (scale).

cup = stats.norm(loc=150, scale=5)           # tea machine: 150 ml, sd 5
print("P(cup < 140 ml) =", round(cup.cdf(140), 4))
print("P(145 to 155)   =", round(cup.cdf(155) - cup.cdf(145), 4))
print("P(140 to 160)   =", round(cup.cdf(160) - cup.cdf(140), 4))
print("P(135 to 165)   =", round(cup.cdf(165) - cup.cdf(135), 4))
P(cup < 140 ml) = 0.0228
P(145 to 155)   = 0.6827
P(140 to 160)   = 0.9545
P(135 to 165)   = 0.9973

P(cup < 140 ml) = 2.3% (the red area under the bell in the figure). Within 1, 2 and 3 standard deviations of the mean: 68.3%, 95.4%, 99.7%.

The standard normal and the 68-95-99.7 rule

Subtract the mean and divide by the standard deviation (the z-score, episode 36) and any normal becomes the standard normal: mean 0, standard deviation 1.

cups = rng.normal(150, 5, 100_000)
z = (cups - 150) / 5                         # standard normal: mean 0, sd 1
for k in [1, 2, 3]:
    print(f"within {k} sd: {np.mean(abs(z) < k):.3f}")
within 1 sd: 0.682
within 2 sd: 0.954
within 3 sd: 0.997

The empirical rule (68-95-99.7 rule): for any normal distribution, about 68% of values lie within 1 sd of the mean, 95% within 2, 99.7% within 3. A cup more than 3 sd short (below 135 ml) happens about once in 740 cups (P = 0.00135). The rule is only for (roughly) normal data; skewed data breaks it.

ppf: the inverse CDF

The CDF turns a value into a probability. The percent point function (ppf), the inverse CDF, turns a probability into a value: "below which volume are the worst 5% of cups?"

cup = stats.norm(150, 5)
print("bottom 5% of cups are below", round(cup.ppf(0.05), 1), "ml")
print("check: cdf(141.8) =", round(cup.cdf(141.8), 3))
print("standard normal ppf(0.975) =", round(stats.norm.ppf(0.975), 2))
bottom 5% of cups are below 141.8 ml
check: cdf(141.8) = 0.051
standard normal ppf(0.975) = 1.96

ppf(0.05) = 141.8 ml (the 5th percentile). For the standard normal, ppf(0.975) = 1.96: 95% of values lie within ±1.96 sd, the critical value you'll use for confidence intervals in episode 45.

Exponential: time between events

If customers arrive as a Poisson process at 12 per hour (episode 41), the time between arrivals is continuous and follows the exponential distribution with mean 60/12 = 5 minutes. In SciPy, scale is the mean.

gap = stats.expon(scale=5)          # 12 per hour → 5 min apart on average
print("P(next customer within 2 min) =", round(gap.cdf(2), 3))
print("P(wait more than 10 min)      =", round(1 - gap.cdf(10), 3))
print("mean gap:", gap.mean(), "min")
P(next customer within 2 min) = 0.33
P(wait more than 10 min)      = 0.135
mean gap: 5.0 min

P(next customer within 2 min) = 1 − e^(−2/5) = 0.33; P(wait > 10 min) = e^(−10/5) = 0.135. Short gaps are common, long gaps rare. The exponential is memoryless: having already waited 5 minutes doesn't change the chance of waiting 5 more.

Log-normal: skewed amounts

A variable is log-normal when its logarithm is normal. Such values are always positive, mostly moderate, with a long right tail: incomes, prices, order values, file sizes.

bill = rng.lognormal(mean=5.5, sigma=0.6, size=10_000)   # made-up order values
print("bill: mean", round(bill.mean()), " median", round(np.median(bill)),
      " skew", round(stats.skew(bill), 2))
print("log(bill): skew", abs(round(stats.skew(np.log(bill)), 2)))
bill: mean 292  median 243  skew 2.3
log(bill): skew 0.0

Mean 292 > median 243, skew 2.3; after taking the log, skew 0. This is why the log transform (episode 36) makes these columns symmetric.

Is my data normal?

Many methods (t-tests, confidence intervals, regression residual checks) assume roughly normal data. Two checks:

  • A Q-Q plot (quantile-quantile plot) plots each data point's quantile against where it would be in a perfect normal distribution. Straight line = normal. A curve means skew; points peeling away at both ends mean heavy tails.
  • The Shapiro-Wilk test gives a p-value; a small p-value (below 0.05) is evidence the data is not normal. (What a p-value is exactly: episode 46.)
import matplotlib.pyplot as plt
cups = rng.normal(150, 5, 300)
bill = rng.lognormal(5.5, 0.6, 300)
fig, axs = plt.subplots(1, 2, figsize=(12, 4.5))
stats.probplot(cups, plot=axs[0]); axs[0].set_title("cups (normal)")
stats.probplot(bill, plot=axs[1]); axs[1].set_title("bills (log-normal)")
fig.tight_layout(); fig.savefig("qq.png")
print("Shapiro cups: p =", round(stats.shapiro(cups).pvalue, 3))
print("Shapiro bill: p =", f"{stats.shapiro(bill).pvalue:.1e}")
Shapiro cups: p = 0.152
Shapiro bill: p = 2.1e-14

qq.png

The cups hug the line (p = 0.152: no evidence against normality); the bills curve away at the top (p ≈ 2 × 10⁻¹⁴: clearly not normal).

With big data, tests become unreliable or over-sensitive:

cups = rng.normal(150, 5, 20_000)
print("p =", round(stats.shapiro(cups).pvalue, 3))
p = 0.856
/usr/local/lib/python3.13/site-packages/scipy/stats/_axis_nan_policy.py:601: UserWarning: scipy.stats.shapiro: For N > 5000, computed p-value may not be accurate. Current N is 20000.
  res = hypotest_fun_out(*samples, **kwds)

UserWarning: scipy.stats.shapiro: For N > 5000, computed p-value may not be accurate. With huge samples, normality tests also flag tiny, harmless departures. For big data, look at the Q-Q plot and histogram first.

Common mistakes

Code / habit Real error / symptom Why Fix
stats.shapiro(x) with 20,000 values UserWarning: scipy.stats.shapiro: For N > 5000, computed p-value may not be accurate. test designed for smaller samples Q-Q plot, histogram
np.mean(wait == 3) no error; 0.0 exact values have probability 0 ask about a range
stats.uniform(5, 10) for 5-10 no error; 5 to 15 scale is the width uniform(loc=5, scale=5)
reading pdf(x) as a probability no error; values can exceed 1 it's a density cdf(b) - cdf(a)
stats.norm(150, 25) meaning variance 25 no error; sd 25, far too wide scale is the sd, not the variance scale=5
applying 68-95-99.7 to skewed data no error; wrong shares the rule is for normal data check the shape first
stats.expon(5) for mean 5 no error; shifted to start at 5 first argument is loc stats.expon(scale=5)

Mini project: quality control for the tea machine

The machine should pour at least 140 ml. How many of 1,000 cups fail, and what mean setting would bring failures below 0.5%?

from scipy import stats
print("fail now:", round(1000 * stats.norm(150, 5).cdf(140), 1), "cups per 1,000")
target = 140 - 5 * stats.norm.ppf(0.005)          # mean so that P(cup < 140) = 0.5%
print("set the mean to", round(target, 1), "ml")
print("check:", round(stats.norm(target, 5).cdf(140), 4))
fail now: 22.8 cups per 1,000
set the mean to 152.9 ml
check: 0.005

About 23 of 1,000 cups are short today; setting the mean to about 152.9 ml brings failures to 0.5%.

Practice

1. Heights are normal with mean 165 cm, sd 7. What share is taller than 180 cm? What height is the top 10% cutoff?

Solution
from scipy import stats
h = stats.norm(165, 7)
print(round(1 - h.cdf(180), 4), round(h.ppf(0.90), 1))
0.0161 174.0

2. A bus comes every 15 minutes and you arrive at a random time (uniform 0-15). P(wait > 10 min)?

Solution
from scipy import stats
print(round(1 - stats.uniform(loc=0, scale=15).cdf(10), 4))
0.3333

3. Calls arrive at 20 per hour. P(no call in the next 6 minutes)?

Solution
from scipy import stats
print(round(1 - stats.expon(scale=3).cdf(6), 4))     # mean gap = 60/20 = 3 minutes
0.1353

Quick quiz

# Marks ~ normal(mean=60, sd=10)
# Roughly what share scored between 40 and 80?

A. 68% B. 95% C. 99.7% D. 50%

Answer

B. 40 and 80 are 2 sd either side of 60; the empirical rule gives about 95% (stats.norm(60, 10).cdf(80) - stats.norm(60, 10).cdf(40) = 0.954).

Interview questions

  1. Why is P(X = x) zero for a continuous random variable? Infinitely many possible values share a total probability of 1; probability comes from areas over ranges.
  2. PDF vs CDF? PDF: density per unit of x (area = probability); CDF: P(X ≤ x).
  3. State the empirical rule. For normal data, about 68%, 95%, 99.7% within 1, 2, 3 sd of the mean.
  4. What does ppf do? It inverts the CDF: probability → value (quantile).
  5. When would you use an exponential distribution? For waiting times between events of a Poisson process.
  6. How do you check normality? Q-Q plot (straight line), histogram, and a test such as Shapiro-Wilk, remembering that tests are over-sensitive with large n.
  7. What is a log-normal distribution? One whose logarithm is normal; positive and right-skewed, like incomes.

Glossary

Term Meaning Example
Continuous random variable Takes any value in a range waiting time
Probability density function (PDF) Density of probability per unit of x; area under it = probability uniform.pdf(3) = 0.1
Cumulative distribution function (CDF) P(X ≤ x), the area to the left cdf(3) = 0.3
Continuous uniform Every value in a range equally likely auto wait 0-10 min
loc / scale SciPy's location and scale parameters (start/width, mean/sd) norm(loc=150, scale=5)
Normal (Gaussian) distribution Symmetric bell curve set by mean and sd cup volumes
Standard normal Normal with mean 0, sd 1; z-scores stats.norm()
Empirical rule (68-95-99.7) Shares within 1, 2, 3 sd for normal data 95% within ±2 sd
Percent point function (ppf) / inverse CDF Probability → value ppf(0.05) = 141.8 ml
Quantile / percentile Value below which a given share lies 5th percentile
Exponential distribution Time between Poisson events gaps between customers
Memoryless Past waiting doesn't change future waiting exponential
Log-normal distribution log of the variable is normal; right-skewed order values
Q-Q plot Data quantiles vs normal quantiles; straight = normal stats.probplot
Shapiro-Wilk test Test of normality; small p = evidence of non-normality stats.shapiro
Normality test Any test of whether data is normal Shapiro-Wilk

Summary

  • For continuous variables, exact values have probability 0; probability = area under the PDF; the CDF gives the area to the left.
  • SciPy's loc/scale: uniform = start/width, normal = mean/sd, exponential: scale = mean.
  • The normal curve and the 68-95-99.7 rule; z-scores give the standard normal.
  • ppf inverts the CDF (1.96 for 97.5%).
  • Exponential = gaps between Poisson events; log-normal = log is normal (skewed amounts).
  • Check normality with a Q-Q plot first, Shapiro-Wilk second (warning above N = 5000).
  • Next episode: sampling methods and sampling bias.