These are the notes for Data Analytics with Python, episode 36. You can describe a column with a centre (episode 34) and a spread (episode 35). Two numbers still cannot show the shape of the data. Today: the distribution, frequency distributions and relative frequency, modality (unimodal, bimodal), skewness (left and right skew), kurtosis and heavy tails, the z-score and standardisation, min-max normalisation, and the log transform. Every term is defined the first time it appears and collected in the Glossary.
Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, SciPy 1.18.1, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its last line, as in the video.
Distribution and frequency tables
The distribution of a variable is the full picture of which values occur and how often. The simplest way to see it is to count each value.
import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
rng = np.random.default_rng(36)
pune = pd.Series(rng.choice([1, 2, 3, 4], 60, p=[0.5, 0.25, 0.15, 0.1]))
nashik = pd.Series(rng.choice([1, 2, 3, 4], 400, p=[0.3, 0.3, 0.25, 0.15]))
counts = pd.DataFrame({"Pune": pune.value_counts(),
"Nashik": nashik.value_counts()}).sort_index()
counts.index.name = "cups"
print(counts)
Pune Nashik
cups
1 37 125
2 11 118
3 8 96
4 4 61
A table of each value with its count is a frequency distribution (frequency table); the count is the frequency. For a continuous number (bills, minutes) you first group values into ranges, bins, with pd.cut (episode 17) and count per bin; a histogram is a drawn frequency table.
Comparing these counts between stalls is unfair: Nashik had almost 7× as many orders. Divide each count by the total to get the relative frequency, the share of observations:
rel = pd.DataFrame({"Pune": pune.value_counts(normalize=True),
"Nashik": nashik.value_counts(normalize=True)}).sort_index()
rel.index.name = "cups"
print(rel.round(2))
print(rel.sum().round(2).tolist())
Pune Nashik
cups
1 0.62 0.31
2 0.18 0.30
3 0.13 0.24
4 0.07 0.15
[1.0, 1.0]
62% of Pune's orders are single cups vs 31% at Nashik. Relative frequencies always add up to 1 (100%). normalize=True works in value_counts, and pd.crosstab(..., normalize=...) does the same for two columns.
Shape 1: modality
Real-life example: crowds at a station. One crowd around one platform = unimodal (one peak, one mode). Two crowds at two platforms = bimodal (two peaks). Two humps usually mean two groups are mixed (Mumbai's regular orders and office lunches in episode 33): split the data by the group and look again. Three or more peaks = multimodal.
Shape 2: skewness
import matplotlib.pyplot as plt
rng = np.random.default_rng(36)
delivery = pd.Series(rng.gamma(2.5, 8, 1000) + 10).round() # minutes
height = pd.Series(rng.normal(165, 7, 1000)).round() # cm
marks = pd.Series(100 - rng.gamma(2.5, 5, 1000)).round() # easy exam
data = {"delivery": delivery, "height": height, "marks": marks}
fig, axs = plt.subplots(1, 3, figsize=(13, 4))
for ax, (name, s) in zip(axs, data.items()):
ax.hist(s, bins=30, color="#9aa5b8")
ax.axvline(s.mean(), color="#e4572e", lw=3)
ax.axvline(s.median(), color="#2a9d8f", lw=3)
ax.set_title(f"{name}: skew {s.skew():.2f}")
print(f"{name:9} mean {s.mean():6.1f} median {s.median():6.1f} skew {s.skew():5.2f}")
fig.tight_layout(); fig.savefig("skew.png")
delivery mean 29.8 median 28.0 skew 1.19
height mean 164.8 median 165.0 skew -0.01
marks mean 87.7 median 89.0 skew -1.27

(Red = mean, green = median.)
- Right-skewed (positively skewed): a long tail to the right; mean > median. Delivery times, incomes, prices.
- Symmetric: both sides mirror each other; mean ≈ median; skew ≈ 0. Heights.
- Left-skewed (negatively skewed): a long tail to the left; mean < median. Marks in an easy exam, age at retirement.
The skew is named after the side the tail points to, not the side of the hump. Skewness is the number for the lean: 0 symmetric, positive = right tail, negative = left tail. Rough guide: |skew| < 0.5 fairly symmetric, 0.5–1 moderate, > 1 strong. pandas' .skew() gives the sample (bias-adjusted) skewness.
Shape 3: kurtosis and heavy tails
rng = np.random.default_rng(36)
calm = pd.Series(rng.normal(60, 5, 2000)) # delivery minutes
rare = rng.random(2000) < 0.04 # 4%: stuck or early
wild = pd.Series(np.where(rare, rng.normal(60, 20, 2000), rng.normal(60, 3.1, 2000)))
for name, s in [("calm", calm), ("wild", wild)]:
print(f"{name}: mean {s.mean():.1f} std {s.std():.1f} "
f"max {s.max():.0f} kurtosis {s.kurt():.1f}")
fig, ax = plt.subplots(figsize=(11, 4.2))
ax.hist([calm, wild], bins=60, range=(0, 120), label=["calm", "wild"])
ax.set_yscale("log"); ax.legend(); ax.set_xlabel("minutes")
fig.tight_layout(); fig.savefig("tails.png")
calm: mean 59.9 std 4.9 max 76 kurtosis -0.1
wild: mean 60.1 std 5.5 max 111 kurtosis 27.7

Same mean, nearly the same standard deviation, but "wild" has rare, extreme values far from the centre: heavy tails (fat tails). The y axis is on a log scale (set_yscale("log")) so bars with 1–2 values are still visible. Kurtosis measures tail heaviness. pandas' .kurt() returns excess kurtosis: 0 for a normal (bell-shaped) distribution, positive when extremes are more common than in a bell curve, negative when tails are lighter. Heavy tails matter wherever the rare day is the expensive one: delivery promises, stock returns, insurance claims, server response times.
z-scores
Real-life example. Asha scored 95 in Maths (hard paper: class mean 70, std 10) and 98 in English (easy paper: class mean 92, std 3). Which is the better result? Raw marks are on different scales. Measure how far above the class she was in standard deviations: the z-score, z = (x − mean) / std.
import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
from scipy import stats
asha = {"Maths": 95, "English": 98}
klass = {"Maths": (70, 10), "English": (92, 3)} # (mean, std) of the class
for sub, (mean, std) in klass.items():
z = (asha[sub] - mean) / std
print(f"{sub:8} mark {asha[sub]} mean {mean} std {std} z = {z:.1f}")
Maths mark 95 mean 70 std 10 z = 2.5
English mark 98 mean 92 std 3 z = 2.0
Maths is her stronger result. z = 0 means exactly average, +1 one std above, −2 two std below. In episode 23, |z| > 3 flagged outliers.
Standardisation applies this to a whole column: subtract its mean, divide by its std. The result always has mean 0 and std 1.
bills = pd.Series([80, 100, 120, 90, 110, 300])
z = (bills - bills.mean()) / bills.std()
print(z.round(2).tolist())
print("mean", abs(round(z.mean(), 2)), "std", round(z.std(), 2))
print("scipy:", stats.zscore(bills).round(2).tolist())
[-0.64, -0.4, -0.16, -0.52, -0.28, 2.01]
mean 0.0 std 1.0
scipy: [-0.71, -0.44, -0.18, -0.57, -0.31, 2.2]
SciPy's numbers differ because stats.zscore uses the population std (ddof=0) and pandas .std() the sample std (ddof=1), episode 35. Pick one per project. (abs(...) only hides a printed -0.0, a tiny negative rounding leftover.) Standardised columns are what many machine-learning models expect (scikit-learn's StandardScaler does exactly this with ddof=0).
Min-max normalisation
Min-max normalisation squeezes values into 0–1: (x − min) / (max − min). Useful for combining columns on different scales into one score, or for data with known limits (pixel brightness 0–255, marks out of 100).
price = pd.Series([80, 100, 120, 90, 110])
mm = (price - price.min()) / (price.max() - price.min())
print(mm.tolist())
price2 = pd.concat([price, pd.Series([5000])], ignore_index=True)
mm2 = (price2 - price2.min()) / (price2.max() - price2.min())
print(mm2.round(3).tolist())
[0.0, 0.5, 1.0, 0.25, 0.75]
[0.0, 0.004, 0.008, 0.002, 0.006, 1.0]
One outlier squashes everything else near 0, because min-max (like the range) depends only on the two extremes. Prefer standardisation, or cap outliers first (episode 23). "Normalisation" and "standardisation" are used loosely in the wild; here standardise = z-scores, normalise = min-max.
Log transform
Strongly right-skewed data (salaries, prices, website traffic) has a few huge values that squash everything else. The logarithm counts multiplying steps: log(1,000) to log(10,000) is the same step as log(10,000) to log(100,000). Applying it to every value is a log transform.
rng = np.random.default_rng(36)
salary = pd.Series(rng.lognormal(10.3, 0.6, 1000).round(-2))
logged = np.log(salary)
print("skew before:", round(salary.skew(), 2))
print("skew after :", round(logged.skew(), 2))
fig, axs = plt.subplots(1, 2, figsize=(12, 4))
axs[0].hist(salary, bins=50); axs[0].set_title("salary")
axs[1].hist(logged, bins=50); axs[1].set_title("np.log(salary)")
fig.tight_layout(); fig.savefig("log.png")
skew before: 1.74
skew after : -0.07

np.log is the natural log (base e ≈ 2.718); np.log10 uses base 10. The shape result is the same. To get back to the original units, use np.exp.
The log of zero (and of negative numbers) does not exist:
tips = pd.Series([0, 10, 20, 50, 500])
print(np.log(tips).tolist())
[-inf, 2.302585092994046, 2.995732273553991, 3.912023005428146, 6.214608098422191]
/usr/local/lib/python3.13/site-packages/pandas/core/arraylike.py:402: RuntimeWarning: divide by zero encountered in log
result = getattr(ufunc, method)(*inputs, **kwargs)
Use np.log1p (log of 1 + x), which maps 0 to 0, and np.expm1 to undo it:
print(np.log1p(tips).round(2).tolist())
print(np.expm1(np.log1p(tips)).round().tolist())
[0.0, 2.4, 3.04, 3.93, 6.22]
[0.0, 10.0, 20.0, 50.0, 500.0]
Common mistakes
| Code / habit | Real error / symptom | Why | Fix |
|---|---|---|---|
np.log(tips) with zeros |
RuntimeWarning: divide by zero encountered in log, value -inf |
log(0) is undefined | np.log1p(tips) |
np.log of negative values |
RuntimeWarning: invalid value encountered in log, value nan |
log of a negative number is undefined | shift the data, or don't log it |
comparing value_counts() of different-size groups |
no error; unfair comparison | bigger group, bigger counts | value_counts(normalize=True) |
| calling a hump on the left "left-skewed" | no error; wrong name | skew is named after the tail | tail right = right skew |
stats.zscore(x) vs pandas formula |
no error; slightly different values | ddof 0 vs 1 | choose one and be consistent |
| min-max with an outlier | no error; everything near 0 | depends only on min and max | standardise, or cap first |
| trusting mean ± std on heavy-tailed data | no error; rare extremes surprise you | std can't see tail weight | check .kurt(), percentiles, a log-scale histogram |
Mini project: a fair "best item" score
Price, rating and number of orders are on different scales. Check their shapes, log-transform the skewed one, standardise all three, and combine into one score (good rating, many orders, low price).
import numpy as np, pandas as pd
rng = np.random.default_rng(3)
df = pd.DataFrame({"price": rng.lognormal(5, 0.8, 500).round(),
"rating": rng.normal(4, 0.5, 500).clip(1, 5).round(1),
"orders": rng.poisson(30, 500)})
print(pd.DataFrame({"skew": df.skew(), "kurt": df.kurt()}).round(2))
df["log_price"] = np.log1p(df["price"])
cols = ["log_price", "rating", "orders"]
z = (df[cols] - df[cols].mean()) / df[cols].std()
df["score"] = z["rating"] + z["orders"] - z["log_price"] # good, popular, cheap
print(df.nlargest(3, "score").round(2))
skew kurt
price 3.57 21.84
rating -0.22 -0.01
orders 0.21 0.09
price rating orders log_price score
440 36.0 5.0 37 3.61 5.06
220 27.0 4.1 43 3.33 4.60
68 26.0 4.3 40 3.30 4.55
Price is strongly right-skewed with heavy tails, so it is logged before standardising; otherwise a few very expensive items would dominate the score. (rng.poisson draws counts; you will meet the Poisson distribution in episode 41.)
Practice
1. Daily spending for 10 days: 200, 150, 300, 250, 180, 220, 2600, 190, 210, 240. Which way is it skewed? Which average would you report?
Solution
import pandas as pd
spend = pd.Series([200, 150, 300, 250, 180, 220, 2600, 190, 210, 240]) # daily spend, rupees
print("mean", spend.mean(), "median", spend.median(), "skew", round(spend.skew(), 2))
mean 454.0 median 215.0 skew 3.14
Strongly right-skewed (one 2,600 day); report the median, 215.
2. Standardise the class marks 55, 62, 70, 48, 81, 66, 74, 59. Who is more than one std above the mean?
Solution
import pandas as pd
marks = pd.Series([55, 62, 70, 48, 81, 66, 74, 59])
z = (marks - marks.mean()) / marks.std()
print(z.round(2).tolist())
print("who is above +1:", marks[z > 1].tolist())
[-0.88, -0.22, 0.53, -1.54, 1.56, 0.15, 0.9, -0.5]
who is above +1: [81]
3. Min-max normalise the ratings 3.2, 4.5, 4.9, 2.8, 4.0.
Solution
import pandas as pd
rating = pd.Series([3.2, 4.5, 4.9, 2.8, 4.0])
mm = (rating - rating.min()) / (rating.max() - rating.min())
print(mm.round(3).tolist())
[0.19, 0.81, 1.0, 0.0, 0.571]
4. Video views: 0, 3, 12, 40, 150, 900, 25000. Reduce the skew.
Solution
import numpy as np, pandas as pd
views = pd.Series([0, 3, 12, 40, 150, 900, 25000])
print("skew before:", round(views.skew(), 2))
print("skew after :", round(np.log1p(views).skew(), 2))
skew before: 2.64
skew after : 0.68
log1p because of the 0.
Quick quiz
# Ravi: 82 in Physics (class mean 70, std 6)
# Neha: 90 in Chemistry (class mean 84, std 4)
z = (x - mean) / std
Who did better relative to their class? A. Neha: 90 > 82 B. Ravi: z = 2.0 vs 1.5 C. Equal D. Can't compare different subjects
Answer
B. Ravi. (82 − 70) / 6 = 2.0; (90 − 84) / 4 = 1.5.
Interview questions
- What is a distribution? Which values a variable takes and how often.
- Frequency vs relative frequency? Count vs count ÷ total; use relative frequency to compare groups of different sizes.
- What does a bimodal distribution suggest? Often two subgroups mixed together; segment the data.
- Right vs left skew, and where are mean and median? Right skew: long right tail, mean > median. Left skew: long left tail, mean < median.
- What does kurtosis measure? What is excess kurtosis? Tail heaviness; excess kurtosis = kurtosis − 3, so a normal distribution scores 0. pandas
.kurt()returns excess kurtosis. - What is a z-score? Distance from the mean in standard deviations: (x − mean) / std.
- Standardisation vs min-max normalisation? Standardisation → mean 0, std 1, copes better with outliers; min-max → range 0–1, sensitive to outliers, good for bounded data.
- Why log-transform data? To reduce right skew, compress large values, and turn multiplicative effects into additive ones.
- How do you log-transform data containing zeros?
np.log1p(log(1 + x)); invert withnp.expm1. - Why might
scipy.stats.zscoreand a pandas z-score differ? ddof 0 vs ddof 1.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Distribution | Which values occur and how often | cups per order |
| Frequency | How many times a value occurs | 37 single-cup orders |
| Frequency distribution / table | Each value (or bin) with its frequency | value_counts() |
| Bin | A range of values grouped together | 20–30 minutes |
| Relative frequency | Frequency ÷ total; a share | 0.62 |
| Modality | Number of peaks | uni-, bi-, multimodal |
| Unimodal | One peak | heights |
| Bimodal | Two peaks, often two mixed groups | Mumbai orders |
| Tail | The thin end of a distribution far from the peak | very late deliveries |
| Skewness | Number for how lopsided a distribution is | s.skew() → 1.19 |
| Right (positive) skew | Long tail to the right; mean > median | delivery times |
| Left (negative) skew | Long tail to the left; mean < median | easy-exam marks |
| Symmetric | Both sides mirror each other; skew ≈ 0 | heights |
| Kurtosis | Measure of tail heaviness | s.kurt() |
| Excess kurtosis | Kurtosis − 3; 0 for a normal distribution | calm −0.1, wild 27.7 |
| Heavy (fat) tails | Extreme values more common than in a bell curve | "wild" deliveries |
| Normal distribution | The symmetric bell-shaped curve (episode 42) | heights |
| Log scale | Axis where equal steps are ×10 | set_yscale("log") |
| z-score | (x − mean) / std | Maths 2.5 |
| Standardisation | Converting a column to z-scores; mean 0, std 1 | (s - s.mean()) / s.std() |
| Min-max normalisation | (x − min) / (max − min); range 0–1 | prices → 0…1 |
| Logarithm | The power you raise a base to; counts multiplying steps | log10(1000) = 3 |
| Log transform | Applying log to every value | np.log(salary) |
np.log1p / np.expm1 |
log(1 + x) and its inverse e^x − 1 | log1p(0) = 0 |
Summary
- A distribution shows which values occur and how often; frequency tables count, relative frequencies compare fairly.
- Shape has three parts: modality (how many humps), skewness (which way the tail points; mean pulled toward it) and kurtosis (how heavy the tails are).
- A z-score puts any value on a "standard deviations from the mean" scale; standardisation gives mean 0, std 1.
- Min-max scales to 0–1 but is ruined by outliers.
- The log transform un-skews right-skewed data; use
log1pwhen there are zeros. - Next episode: exploratory data analysis, a checklist for exploring any new dataset one variable at a time.