SyntaxPilot
Courses/Data Analytics with Python/Lesson 43
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Data Analytics with Python · Lesson 43 of 90

Sampling methods and sampling bias

parameterstatisticsampling framesimple random samplingstratified samplingcluster samplingsystematic samplingconvenience samplingsampling biasselection biassurvivorship biasnon-response biassampling in pandas (sample, stratified sample)

These are the notes for Data Analytics with Python, episode 43. Until now we had the whole dataset. In real life you almost never do: you can't call every customer, test every cup or ask every voter. So you measure a few and talk about all of them. These notes cover the population and the sample, parameters and statistics, four ways to sample (simple random, stratified, cluster, systematic), the sampling frame, and the biases that fool even huge surveys: convenience sampling, selection bias, non-response bias and survivorship bias. Every term is defined the first time it appears and collected in the Glossary.

Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. TiffinBox is a made-up tiffin delivery app and all its data is made up for teaching. Every program starts with these lines, which build TiffinBox's 100,000 customers:

import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
rng = np.random.default_rng(43); N = 100_000    # every TiffinBox customer
zone = rng.choice(["North", "South", "East", "West"], N, p=[.4, .3, .2, .1])
avg = pd.Series(zone).map({"North": 3, "South": 4, "East": 5, "West": 12})
orders = rng.poisson(avg)                         # orders a month
pop = pd.DataFrame({"zone": zone, "orders": orders,
    "spend": orders * rng.normal(180, 30, N).round(),
    "society": pd.Series(zone).str[0] + rng.integers(1, 101, N).astype(str),
    "rating": rng.choice([1, 2, 3, 4, 5], N, p=[.1, .15, .25, .3, .2])})

Each customer has a zone (40% North, 30% South, 20% East, 10% West), monthly orders (West customers order about 12 times a month, the others 3-5), monthly spend in rupees, a society (one of 100 housing societies per zone, like N17) and a rating of 1-5 stars. Because we built the population ourselves, we can always print the true answer and check every method against it.

The problem: the obvious shortcut fails

The boss asks: what is our customers' average monthly spend? Calling 100,000 people takes weeks, so the obvious shortcut is to survey the 500 people who opened the app today, the most active customers.

print("true mean spend (all 100k):", round(pop.spend.mean()))
today = pop.sort_values("orders", ascending=False).head(500)  # opened app today
print("survey of today's 500 users:", round(today.spend.mean()))
true mean spend (all 100k): 826
survey of today's 500 users: 3511

The survey says ₹3,511; the truth is ₹826. More than four times too high, and it would be wrong in the same direction every time you repeat it. Why: who got into the sample depended on the very thing we measured. Heavy users open the app more, and heavy users spend more.

  • Convenience sampling: surveying whoever is easiest to reach (today's app users, your friends, people at one mall).
  • Selection bias: the way units get into the sample depends on the thing you measure, so the sample is not like the population.

Real-life example: tasting the dal

Your mother checks the salt in a whole pot of dal by tasting one spoonful. But she stirs first: without stirring, the spoon only tastes the top layer.

In the kitchen In data
the whole pot population: every unit you want to know about
one spoonful sample: the units you actually measure
stirring before tasting random selection: every unit gets a fair chance

Four words to get right

  • Population: every unit you care about (all 100,000 customers). A unit can be a person, an order, a shop, a day.
  • Sample: the part of the population you actually measure (500 customers).
  • Parameter: a number that describes the population, like the true mean spend ₹826. You usually never see it.
  • Statistic: the same number computed from the sample. We use a statistic to estimate a parameter.
  • Memory trick: Population → Parameter, Sample → Statistic.
  • Sampling bias: the sampling method itself favours some units, so the estimate is off in one direction. A bigger sample does not fix bias.

Simple random sampling

In a simple random sample (SRS) every unit has the same chance of being picked, and every group of n units is equally likely. In pandas, DataFrame.sample(n=...) draws one. random_state fixes the random shuffle, so you get the same rows every run (reproducibility, from episode 10).

srs = pop.sample(n=500, random_state=1)       # simple random sample
print("parameter (all 100k):", round(pop.spend.mean()))
print("statistic (500 rows):", round(srs.spend.mean()))
for s in [2, 3, 4]:
    m = pop.sample(500, random_state=s).spend.mean()
    print(f"another sample, seed {s}:", round(m))
parameter (all 100k): 826
statistic (500 rows): 871
another sample, seed 2: 851
another sample, seed 3: 866
another sample, seed 4: 798

Every random sample gives a slightly different answer: sampling variability (also called sampling error). Random samples miss by a few dozen rupees, sometimes up, sometimes down. The biased survey missed by ₹2,700, always up.

Random error (sampling variability) Bias
direction up or down, at random always the same way
bigger sample shrinks it does not help
fix larger n a better method

Useful sample arguments: n (number of rows) or frac (share of rows), random_state (seed), replace (allow the same row twice, default False), weights (unequal chances).

Stratified sampling

The West zone is only 10% of customers, but they order about three times as often as the rest. A random 500 can get too few or too many West customers by luck, and that swings the estimate.

  • A stratum (plural strata) is a group of the population that you sample inside, like a zone, a city, a class or a department.
  • Stratified sampling: split the population into strata, then take a random sample inside each stratum. With the same share from every stratum (proportional allocation), the sample keeps the population's mix exactly.
print(pop.zone.value_counts(normalize=True).round(2).to_dict())
one = pop.groupby("zone").sample(frac=0.005, random_state=1)   # 0.5% of each zone
print(one.zone.value_counts().to_dict())
srs = [pop.sample(500, random_state=s).spend.mean() for s in range(200)]
strat = [pop.groupby("zone").sample(frac=0.005, random_state=s).spend.mean()
         for s in range(200)]
print("200 random samples    :", round(min(srs)), "to", round(max(srs)))
print("200 stratified samples:", round(min(strat)), "to", round(max(strat)))
{'North': 0.4, 'South': 0.3, 'East': 0.2, 'West': 0.1}
{'North': 200, 'South': 151, 'East': 99, 'West': 50}
200 random samples    : 746 to 942
200 stratified samples: 783 to 866

groupby("zone").sample(frac=0.005) takes half a percent of every zone: 200, 151, 99, 50. Over 200 repeats, stratified estimates stay in a much tighter range than simple random ones of the same size. Stratifying helps most when the strata really differ in what you measure (here, West spends far more).

If you take the same number from every stratum (groupby(...).sample(n=10)), small strata are over-represented, so you must weight each stratum's mean by its population share when you combine them (see the mini project).

Cluster sampling

The survey is door to door, and visiting 500 homes all over the city is expensive. The customers live in 400 housing societies.

  • A cluster is a natural group of units that sit together, like a housing society, a school, a village or a branch.
  • Cluster sampling: randomly pick a few clusters and measure everyone inside them.
socs = pd.Series(pop.society.unique())
print("housing societies:", len(socs))
clus = pop[pop.society.isin(socs.sample(20, random_state=1))]  # EVERYONE in 20
print("surveyed:", len(clus), "people, in just 20 societies")
est = [pop[pop.society.isin(socs.sample(20, random_state=s))].spend.mean()
       for s in range(100)]
srs = [pop.sample(len(clus), random_state=s).spend.mean() for s in range(100)]
print("100 cluster samples:", round(min(est)), "to", round(max(est)))
print("100 random samples :", round(min(srs)), "to", round(max(srs)))
housing societies: 400
surveyed: 4832 people, in just 20 societies
100 cluster samples: 662 to 1055
100 random samples : 800 to 844

4,832 people from only 20 places: cheap to collect. But the estimates swing from ₹662 to ₹1,055, while random samples of the same size stay between ₹800 and ₹844. Why: people inside one cluster are alike (here every society sits in one zone), so 20 societies carry much less information than 4,832 independent people. Cluster sampling trades precision for cost.

Stratified Cluster
groups strata clusters
what you sample some units from every group all units from some groups
best when groups differ from each other groups are alike, and reaching them is costly
effect on precision better worse (for the same n)

The three methods in one picture

import matplotlib.pyplot as plt
true = pop.spend.mean()
socs = pd.Series(pop.society.unique())
r = {"simple random\n(500)": [pop.sample(500, random_state=s).spend.mean() for s in range(200)],
     "stratified\n(500)": [pop.groupby("zone").sample(frac=.005, random_state=s).spend.mean() for s in range(200)],
     "cluster\n(20 societies)": [pop[pop.society.isin(socs.sample(20, random_state=s))].spend.mean() for s in range(200)]}
fig, ax = plt.subplots(figsize=(13, 4.6))
for i, (k, v) in enumerate(r.items()):
    ax.scatter(v, np.full(len(v), i) + rng.uniform(-.18, .18, len(v)), s=14, alpha=.55)
ax.axvline(true, color="#e4572e", lw=2.5, label=f"true mean {true:.0f}")
conv = pop.sort_values("orders", ascending=False).head(500).spend.mean()
ax.set_yticks(range(3), list(r)); ax.set_xlabel("estimated mean monthly spend (Rs)")
ax.set_title(f"200 samples of each kind  ·  the 'today's users' survey said {conv:.0f}: far off the right edge")
ax.legend(loc="upper right"); fig.tight_layout(); fig.savefig("spread.png", dpi=110)

spread.png

Each dot is one sample's estimate; the red line is the truth. All three random methods scatter around the truth (they are unbiased). Stratified hugs it, cluster spreads widest. The convenience survey (₹3,511) is so far to the right it is off the chart: a biased method lands on one side every time.

Systematic sampling

  • Systematic sampling: from a list, take every k-th row (every 24th, every 100th), usually from a random start. In pandas: df.iloc[start::k].

It is easy and spreads the sample across the list, but it breaks when the list has a repeating pattern whose length matches k. Thirty days of hourly sales, with a lunch rush from 12 to 2:

hours = np.tile(np.arange(24), 30)              # 30 days of hourly sales
lunch = (hours >= 12) & (hours <= 14)
sales = pd.DataFrame({"hour": hours,
                      "orders": rng.poisson(np.where(lunch, 90, 15))})
print("true mean orders per hour:", round(sales.orders.mean(), 1))
every24 = sales.iloc[5::24]                    # systematic: every 24th row
print("every 24th row, hours:", every24.hour.unique())
print("every 24th row estimate:", round(every24.orders.mean(), 1))
print("every 7th row estimate :", round(sales.iloc[5::7].orders.mean(), 1))
true mean orders per hour: 24.3
every 24th row, hours: [5]
every 24th row estimate: 14.6
every 7th row estimate : 24.4

Every 24th row is always 5 AM, so the estimate (14.6) misses the lunch rush completely. Every 7th row cycles through all the hours and gets 24.4. Use systematic sampling only on lists with no pattern, or pick a step that cannot line up with the pattern.

The sampling frame: the 1936 Literary Digest poll

  • Sampling frame: the actual list you draw the sample from (a customer database, a phone directory, a voter roll). If the frame misses part of the population, no sampling method can fix it. This is also called coverage error.

A real story. In 1936 the American magazine The Literary Digest tried to predict the US presidential election. It mailed about 10 million ballots to names taken mostly from telephone directories and car registrations (and club membership lists). About 2.4 million came back, roughly one in four. It predicted that the challenger Alf Landon would win about 57% of the vote. Franklin D. Roosevelt won with about 62%.

Two things went wrong: 1. The frame: during the Great Depression, people with phones and cars were wealthier than average and leaned towards Landon. 2. Non-response: Landon supporters were more likely to send their ballots back.

Millions of replies did not help, because sample size cannot fix a biased frame or biased replies. (Sources: Matthew Salganik, Bit by Bit, section 3.3.1, bitbybitbook.com; Emory University, "Famous Statistical Blunders in History".)

Non-response bias

  • Non-response bias: the people who answer differ from the people who don't, so the answers you get are not like the population.

TiffinBox sends a rating survey to all 100,000 customers, and happier customers are more likely to reply (a 5-star customer replies with chance 0.5, a 1-star customer with 0.1):

replies = rng.random(N) < pop.rating / 10       # happy customers reply more
print("reply rate:", round(replies.mean(), 2))
print("true mean rating  :", round(pop.rating.mean(), 2))
print("replies' mean     :", round(pop[replies].rating.mean(), 2))
reply rate: 0.33
true mean rating  : 3.34
replies' mean     : 3.81

Nobody lied, and everyone was asked, yet the replies say 3.81 stars instead of 3.34. What to do: always report the response rate (the share who replied), compare responders with non-responders on things you already know (zone, orders), send reminders, and be careful with any survey that people choose to answer.

Survivorship bias

A real story. In World War 2, officials studied the bombers that came back from missions and mapped their bullet holes: mostly on the wings, body and tail, with the engines fairly clean. The obvious fix was to add armour where the holes were. The mathematician Abraham Wald, working with the Statistical Research Group at Columbia University, pointed out that the data only contained planes that survived. Planes hit in the engines never came back, so the clean areas on survivors were the fatal ones. His advice: armour the places without holes. (Source: summary of the Statistical Research Group story, yukaichou.com.)

  • Survivorship bias: drawing conclusions only from the units that made it through some filter (survived, stayed open, stayed in the list), while the ones that didn't make it are invisible.
q = rng.normal(3.4, 0.8, 2000).clip(1, 5)       # 2,000 restaurants opened
shops = pd.DataFrame({"rating": q.round(1), "open": q > 3.0})  # weak ones closed
print("all that opened:", round(shops.rating.mean(), 2))
print("still open (the app's list):", round(shops[shops.open].rating.mean(), 2),
      f"({shops.open.sum()} of 2000)")
all that opened: 3.39
still open (the app's list): 3.81 (1370 of 2000)

The app's restaurant list only has the 1,370 survivors, so it looks better (3.81) than reality (3.39). The same trap hides in "successful startups", "top mutual funds of the last 10 years", "customers who stayed for a year" and "dropout founders". Always ask: who is missing from this table, and why?

Bias types at a glance

Bias What goes wrong Example
Selection bias who gets in depends on what you measure surveying today's app users
Convenience sampling a kind of selection bias: easiest to reach asking your friends
Coverage error (bad frame) the list misses part of the population phone directories in 1936
Non-response bias responders differ from non-responders happy customers reply more
Survivorship bias only the survivors are in the data bombers that came back

Common mistakes

Code / habit Real error / symptom Why Fix
pop.sample(200_000) on 100,000 rows ValueError: Cannot take a larger sample than population when 'replace=False' without replacement each row can be picked once smaller n, or replace=True (the bootstrap, episode 45)
surveying today's app users no error; ₹3,511 vs ₹826 convenience sample, selection bias random sample from the full list
sales.iloc[::24] on hourly data no error; always the same hour the step matches the pattern random sample, or a step that can't line up
analysing only replies no error; ratings too high non-response bias report the response rate, compare responders
analysing only survivors no error; looks too good survivorship bias find the ones that dropped out
forgetting random_state no error; different rows every run the shuffle changes set random_state for reproducible work
equal n per stratum, plain mean no error; small strata count too much the sample no longer matches the mix weight each stratum mean by its share

Mini project: a frame that forgot the West zone

TiffinBox's survey list was exported from an old system that never added the West zone. What happens to the estimate, and how does a stratified sample from the full list compare?

frame = pop[pop.zone != "West"]                 # the list forgot the West zone
print("true mean:", round(pop.spend.mean()))
est = [frame.sample(500, random_state=s).spend.mean() for s in range(200)]
print("random samples from the bad frame:", round(min(est)), "to", round(max(est)))
print("average of those estimates:", round(np.mean(est)))
w = pop.zone.value_counts(normalize=True)
fixed = (pop.groupby("zone").sample(frac=0.005, random_state=1)
            .groupby("zone").spend.mean() * w).sum()
print("stratified with every zone, weighted:", round(fixed))
true mean: 826
random samples from the bad frame: 630 to 722
average of those estimates: 680
stratified with every zone, weighted: 809

Even perfectly random sampling from the bad frame averages ₹680: every one of 200 samples is too low, because the high-spending West zone can never be picked. A stratified sample from the complete list, with each zone's mean weighted by its population share, gives ₹809, close to the true ₹826.

Practice

1. A college has 1,000 students in classes A, B and C (50%, 30%, 20%). Draw a simple random sample of 50 and a stratified sample of 5% from each class, and compare the class counts.

Solution
import pandas as pd, numpy as np
rng = np.random.default_rng(7)
students = pd.DataFrame({"cls": rng.choice(["A", "B", "C"], 1000, p=[.5, .3, .2]),
                         "marks": rng.integers(30, 101, 1000)})
srs = students.sample(n=50, random_state=1)
strat = students.groupby("cls").sample(frac=0.05, random_state=1)
print(srs.cls.value_counts().to_dict())
print(strat.cls.value_counts().to_dict())
{'A': 20, 'C': 17, 'B': 13}
{'A': 25, 'B': 15, 'C': 10}

The random sample got class C almost twice its share by luck; the stratified one matches the 50/30/20 mix.

2. Take exactly 10 students from each class.

Solution
import pandas as pd, numpy as np
rng = np.random.default_rng(7)
students = pd.DataFrame({"cls": rng.choice(["A", "B", "C"], 1000, p=[.5, .3, .2]),
                         "marks": rng.integers(30, 101, 1000)})
eq = students.groupby("cls").sample(n=10, random_state=1)    # 10 from EACH class
print(eq.cls.value_counts().to_dict(), len(eq))
{'A': 10, 'B': 10, 'C': 10} 30

Remember to weight each class's mean by its population share (0.5, 0.3, 0.2) when you estimate the overall mean.

3. Draw a systematic sample of every 50th row from a 1,000-row list, with a random start.

Solution
import pandas as pd, numpy as np
rng = np.random.default_rng(3)
df = pd.DataFrame({"id": range(1, 1001)})
k = 50
start = int(rng.integers(0, k))                 # random start between 0 and k-1
sysm = df.iloc[start::k]
print("start:", start, " rows:", len(sysm), " first ids:", sysm.id.head(4).tolist())
start: 40  rows: 20  first ids: [41, 91, 141, 191]

4. Name the bias: a gym studies only members who have stayed for 2 years and concludes that its programme keeps everyone fit.

Answer

Survivorship bias. Members who quit (perhaps because the programme didn't work for them) are missing from the data.

Quick quiz

# 4 departments, very different sizes.
# Randomly pick 2% of EACH department.
sample = students.groupby("dept").sample(frac=0.02)

A. Simple random B. Stratified C. Cluster D. Systematic

Answer

B. Stratified. Each department is a stratum and a random share is taken inside every one. Cluster sampling would pick a few whole departments and survey everyone in them.

Interview questions

  1. Parameter vs statistic? A parameter describes the population (usually unknown); a statistic is computed from a sample and estimates it.
  2. Why can't a bigger sample fix sampling bias? Bias moves every estimate the same way; more data from the same biased method just gives a more precise wrong answer (Literary Digest: 2.4 million replies).
  3. Stratified vs cluster sampling? Stratified samples some units from every group (more precise when groups differ); cluster samples all units from a few groups (cheaper, less precise when units within a cluster are alike).
  4. When does systematic sampling fail? When the list has a periodic pattern that lines up with the step.
  5. What is a sampling frame and why does it matter? The list you sample from; anyone missing from it can never be sampled (coverage error).
  6. Give an example of survivorship bias in business data. Analysing only current customers to explain churn, or only funds that still exist to judge fund performance.
  7. How do you take a stratified sample in pandas? df.groupby("stratum").sample(frac=...) or .sample(n=...) per group.
  8. How do you check for non-response bias? Report the response rate and compare responders with non-responders on known variables.

Glossary

Term Meaning Example
Population Every unit you want to know about all 100,000 TiffinBox customers
Sample The units you actually measure 500 customers
Unit One member of the population one customer, one order
Parameter A number describing the population true mean spend ₹826
Statistic The same number computed from a sample sample mean ₹871
Estimate Using a statistic as a guess for a parameter "about ₹871"
Sampling frame The list you actually draw the sample from customer database, phone directory
Coverage error The frame misses part of the population West zone missing from the list
Simple random sampling (SRS) Every unit and every group of n units equally likely pop.sample(n=500)
random_state Seed that fixes the random choice so it repeats sample(500, random_state=1)
Sampling variability Different samples give different statistics 871, 851, 866, 798
Stratum / strata A group of the population you sample inside a zone
Stratified sampling Random sample inside every stratum groupby("zone").sample(frac=0.005)
Proportional allocation Same share from every stratum 0.5% of each zone
Cluster A natural group of units that sit together a housing society
Cluster sampling Pick some clusters, measure everyone in them 20 societies, 4,832 people
Systematic sampling Every k-th unit from a list, random start df.iloc[start::k]
Convenience sampling Measuring whoever is easiest to reach today's app users
Sampling bias The method favours some units, so estimates lean one way ₹3,511 vs ₹826
Selection bias Who gets in depends on what you measure heavy users open the app more
Non-response bias Responders differ from non-responders happy customers reply more
Response rate Share of people asked who replied 0.33
Survivorship bias Only units that passed a filter are in the data bombers that came back
Random error vs bias Random error scatters and shrinks with n; bias leans one way and doesn't SRS vs convenience
replace Whether a row can be picked more than once replace=False (default)

Summary

  • Population → parameter, sample → statistic. We estimate parameters from statistics.
  • Convenience samples suffer from selection bias; bias doesn't shrink with a bigger sample.
  • Simple random sampling (df.sample) stirs the pot; estimates scatter around the truth (sampling variability).
  • Stratified sampling (groupby().sample) keeps every group in proportion and gives steadier estimates.
  • Cluster sampling is cheap but less precise; systematic sampling breaks on patterned lists.
  • The sampling frame, non-response and survivorship can bias any study, however large: ask who is missing.
  • Next episode: sampling distributions and the central limit theorem: exactly how much a random sample's answer wobbles.