SyntaxPilot
Courses/Data Analytics with Python/Lesson 36
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Data Analytics with Python · Lesson 36 of 90

Descriptive statistics 3: shape, skew and z-scores

distributionfrequency distributionrelative frequencyskewnessleft/right skewkurtosisheavy tailsmodality (unimodal, bimodal)z-scorestandardisationmin-max normalisationlog transform

These are the notes for Data Analytics with Python, episode 36. You can describe a column with a centre (episode 34) and a spread (episode 35). Two numbers still cannot show the shape of the data. Today: the distribution, frequency distributions and relative frequency, modality (unimodal, bimodal), skewness (left and right skew), kurtosis and heavy tails, the z-score and standardisation, min-max normalisation, and the log transform. Every term is defined the first time it appears and collected in the Glossary.

Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, SciPy 1.18.1, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its last line, as in the video.

Distribution and frequency tables

The distribution of a variable is the full picture of which values occur and how often. The simplest way to see it is to count each value.

import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
rng = np.random.default_rng(36)
pune = pd.Series(rng.choice([1, 2, 3, 4], 60, p=[0.5, 0.25, 0.15, 0.1]))
nashik = pd.Series(rng.choice([1, 2, 3, 4], 400, p=[0.3, 0.3, 0.25, 0.15]))
counts = pd.DataFrame({"Pune": pune.value_counts(),
                       "Nashik": nashik.value_counts()}).sort_index()
counts.index.name = "cups"
print(counts)
      Pune  Nashik
cups              
1       37     125
2       11     118
3        8      96
4        4      61

A table of each value with its count is a frequency distribution (frequency table); the count is the frequency. For a continuous number (bills, minutes) you first group values into ranges, bins, with pd.cut (episode 17) and count per bin; a histogram is a drawn frequency table.

Comparing these counts between stalls is unfair: Nashik had almost 7× as many orders. Divide each count by the total to get the relative frequency, the share of observations:

rel = pd.DataFrame({"Pune": pune.value_counts(normalize=True),
                    "Nashik": nashik.value_counts(normalize=True)}).sort_index()
rel.index.name = "cups"
print(rel.round(2))
print(rel.sum().round(2).tolist())
      Pune  Nashik
cups              
1     0.62    0.31
2     0.18    0.30
3     0.13    0.24
4     0.07    0.15
[1.0, 1.0]

62% of Pune's orders are single cups vs 31% at Nashik. Relative frequencies always add up to 1 (100%). normalize=True works in value_counts, and pd.crosstab(..., normalize=...) does the same for two columns.

Shape 1: modality

Real-life example: crowds at a station. One crowd around one platform = unimodal (one peak, one mode). Two crowds at two platforms = bimodal (two peaks). Two humps usually mean two groups are mixed (Mumbai's regular orders and office lunches in episode 33): split the data by the group and look again. Three or more peaks = multimodal.

Shape 2: skewness

import matplotlib.pyplot as plt
rng = np.random.default_rng(36)
delivery = pd.Series(rng.gamma(2.5, 8, 1000) + 10).round()      # minutes
height = pd.Series(rng.normal(165, 7, 1000)).round()             # cm
marks = pd.Series(100 - rng.gamma(2.5, 5, 1000)).round()        # easy exam
data = {"delivery": delivery, "height": height, "marks": marks}
fig, axs = plt.subplots(1, 3, figsize=(13, 4))
for ax, (name, s) in zip(axs, data.items()):
    ax.hist(s, bins=30, color="#9aa5b8")
    ax.axvline(s.mean(), color="#e4572e", lw=3)
    ax.axvline(s.median(), color="#2a9d8f", lw=3)
    ax.set_title(f"{name}: skew {s.skew():.2f}")
    print(f"{name:9} mean {s.mean():6.1f}  median {s.median():6.1f}  skew {s.skew():5.2f}")
fig.tight_layout(); fig.savefig("skew.png")
delivery  mean   29.8  median   28.0  skew  1.19
height    mean  164.8  median  165.0  skew -0.01
marks     mean   87.7  median   89.0  skew -1.27

skew.png

(Red = mean, green = median.)

  • Right-skewed (positively skewed): a long tail to the right; mean > median. Delivery times, incomes, prices.
  • Symmetric: both sides mirror each other; mean ≈ median; skew ≈ 0. Heights.
  • Left-skewed (negatively skewed): a long tail to the left; mean < median. Marks in an easy exam, age at retirement.

The skew is named after the side the tail points to, not the side of the hump. Skewness is the number for the lean: 0 symmetric, positive = right tail, negative = left tail. Rough guide: |skew| < 0.5 fairly symmetric, 0.5–1 moderate, > 1 strong. pandas' .skew() gives the sample (bias-adjusted) skewness.

Shape 3: kurtosis and heavy tails

rng = np.random.default_rng(36)
calm = pd.Series(rng.normal(60, 5, 2000))                       # delivery minutes
rare = rng.random(2000) < 0.04                                  # 4%: stuck or early
wild = pd.Series(np.where(rare, rng.normal(60, 20, 2000), rng.normal(60, 3.1, 2000)))
for name, s in [("calm", calm), ("wild", wild)]:
    print(f"{name}: mean {s.mean():.1f}  std {s.std():.1f}  "
          f"max {s.max():.0f}  kurtosis {s.kurt():.1f}")
fig, ax = plt.subplots(figsize=(11, 4.2))
ax.hist([calm, wild], bins=60, range=(0, 120), label=["calm", "wild"])
ax.set_yscale("log"); ax.legend(); ax.set_xlabel("minutes")
fig.tight_layout(); fig.savefig("tails.png")
calm: mean 59.9  std 4.9  max 76  kurtosis -0.1
wild: mean 60.1  std 5.5  max 111  kurtosis 27.7

tails.png

Same mean, nearly the same standard deviation, but "wild" has rare, extreme values far from the centre: heavy tails (fat tails). The y axis is on a log scale (set_yscale("log")) so bars with 1–2 values are still visible. Kurtosis measures tail heaviness. pandas' .kurt() returns excess kurtosis: 0 for a normal (bell-shaped) distribution, positive when extremes are more common than in a bell curve, negative when tails are lighter. Heavy tails matter wherever the rare day is the expensive one: delivery promises, stock returns, insurance claims, server response times.

z-scores

Real-life example. Asha scored 95 in Maths (hard paper: class mean 70, std 10) and 98 in English (easy paper: class mean 92, std 3). Which is the better result? Raw marks are on different scales. Measure how far above the class she was in standard deviations: the z-score, z = (x − mean) / std.

import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
from scipy import stats
asha = {"Maths": 95, "English": 98}
klass = {"Maths": (70, 10), "English": (92, 3)}   # (mean, std) of the class
for sub, (mean, std) in klass.items():
    z = (asha[sub] - mean) / std
    print(f"{sub:8} mark {asha[sub]}  mean {mean}  std {std}  z = {z:.1f}")
Maths    mark 95  mean 70  std 10  z = 2.5
English  mark 98  mean 92  std 3  z = 2.0

Maths is her stronger result. z = 0 means exactly average, +1 one std above, −2 two std below. In episode 23, |z| > 3 flagged outliers.

Standardisation applies this to a whole column: subtract its mean, divide by its std. The result always has mean 0 and std 1.

bills = pd.Series([80, 100, 120, 90, 110, 300])
z = (bills - bills.mean()) / bills.std()
print(z.round(2).tolist())
print("mean", abs(round(z.mean(), 2)), "std", round(z.std(), 2))
print("scipy:", stats.zscore(bills).round(2).tolist())
[-0.64, -0.4, -0.16, -0.52, -0.28, 2.01]
mean 0.0 std 1.0
scipy: [-0.71, -0.44, -0.18, -0.57, -0.31, 2.2]

SciPy's numbers differ because stats.zscore uses the population std (ddof=0) and pandas .std() the sample std (ddof=1), episode 35. Pick one per project. (abs(...) only hides a printed -0.0, a tiny negative rounding leftover.) Standardised columns are what many machine-learning models expect (scikit-learn's StandardScaler does exactly this with ddof=0).

Min-max normalisation

Min-max normalisation squeezes values into 0–1: (x − min) / (max − min). Useful for combining columns on different scales into one score, or for data with known limits (pixel brightness 0–255, marks out of 100).

price = pd.Series([80, 100, 120, 90, 110])
mm = (price - price.min()) / (price.max() - price.min())
print(mm.tolist())
price2 = pd.concat([price, pd.Series([5000])], ignore_index=True)
mm2 = (price2 - price2.min()) / (price2.max() - price2.min())
print(mm2.round(3).tolist())
[0.0, 0.5, 1.0, 0.25, 0.75]
[0.0, 0.004, 0.008, 0.002, 0.006, 1.0]

One outlier squashes everything else near 0, because min-max (like the range) depends only on the two extremes. Prefer standardisation, or cap outliers first (episode 23). "Normalisation" and "standardisation" are used loosely in the wild; here standardise = z-scores, normalise = min-max.

Log transform

Strongly right-skewed data (salaries, prices, website traffic) has a few huge values that squash everything else. The logarithm counts multiplying steps: log(1,000) to log(10,000) is the same step as log(10,000) to log(100,000). Applying it to every value is a log transform.

rng = np.random.default_rng(36)
salary = pd.Series(rng.lognormal(10.3, 0.6, 1000).round(-2))
logged = np.log(salary)
print("skew before:", round(salary.skew(), 2))
print("skew after :", round(logged.skew(), 2))
fig, axs = plt.subplots(1, 2, figsize=(12, 4))
axs[0].hist(salary, bins=50); axs[0].set_title("salary")
axs[1].hist(logged, bins=50); axs[1].set_title("np.log(salary)")
fig.tight_layout(); fig.savefig("log.png")
skew before: 1.74
skew after : -0.07

log.png

np.log is the natural log (base e ≈ 2.718); np.log10 uses base 10. The shape result is the same. To get back to the original units, use np.exp.

The log of zero (and of negative numbers) does not exist:

tips = pd.Series([0, 10, 20, 50, 500])
print(np.log(tips).tolist())
[-inf, 2.302585092994046, 2.995732273553991, 3.912023005428146, 6.214608098422191]
/usr/local/lib/python3.13/site-packages/pandas/core/arraylike.py:402: RuntimeWarning: divide by zero encountered in log
  result = getattr(ufunc, method)(*inputs, **kwargs)

Use np.log1p (log of 1 + x), which maps 0 to 0, and np.expm1 to undo it:

print(np.log1p(tips).round(2).tolist())
print(np.expm1(np.log1p(tips)).round().tolist())
[0.0, 2.4, 3.04, 3.93, 6.22]
[0.0, 10.0, 20.0, 50.0, 500.0]

Common mistakes

Code / habit Real error / symptom Why Fix
np.log(tips) with zeros RuntimeWarning: divide by zero encountered in log, value -inf log(0) is undefined np.log1p(tips)
np.log of negative values RuntimeWarning: invalid value encountered in log, value nan log of a negative number is undefined shift the data, or don't log it
comparing value_counts() of different-size groups no error; unfair comparison bigger group, bigger counts value_counts(normalize=True)
calling a hump on the left "left-skewed" no error; wrong name skew is named after the tail tail right = right skew
stats.zscore(x) vs pandas formula no error; slightly different values ddof 0 vs 1 choose one and be consistent
min-max with an outlier no error; everything near 0 depends only on min and max standardise, or cap first
trusting mean ± std on heavy-tailed data no error; rare extremes surprise you std can't see tail weight check .kurt(), percentiles, a log-scale histogram

Mini project: a fair "best item" score

Price, rating and number of orders are on different scales. Check their shapes, log-transform the skewed one, standardise all three, and combine into one score (good rating, many orders, low price).

import numpy as np, pandas as pd
rng = np.random.default_rng(3)
df = pd.DataFrame({"price": rng.lognormal(5, 0.8, 500).round(),
                   "rating": rng.normal(4, 0.5, 500).clip(1, 5).round(1),
                   "orders": rng.poisson(30, 500)})
print(pd.DataFrame({"skew": df.skew(), "kurt": df.kurt()}).round(2))
df["log_price"] = np.log1p(df["price"])
cols = ["log_price", "rating", "orders"]
z = (df[cols] - df[cols].mean()) / df[cols].std()
df["score"] = z["rating"] + z["orders"] - z["log_price"]   # good, popular, cheap
print(df.nlargest(3, "score").round(2))
        skew   kurt
price   3.57  21.84
rating -0.22  -0.01
orders  0.21   0.09
     price  rating  orders  log_price  score
440   36.0     5.0      37       3.61   5.06
220   27.0     4.1      43       3.33   4.60
68    26.0     4.3      40       3.30   4.55

Price is strongly right-skewed with heavy tails, so it is logged before standardising; otherwise a few very expensive items would dominate the score. (rng.poisson draws counts; you will meet the Poisson distribution in episode 41.)

Practice

1. Daily spending for 10 days: 200, 150, 300, 250, 180, 220, 2600, 190, 210, 240. Which way is it skewed? Which average would you report?

Solution
import pandas as pd
spend = pd.Series([200, 150, 300, 250, 180, 220, 2600, 190, 210, 240])  # daily spend, rupees
print("mean", spend.mean(), "median", spend.median(), "skew", round(spend.skew(), 2))
mean 454.0 median 215.0 skew 3.14

Strongly right-skewed (one 2,600 day); report the median, 215.

2. Standardise the class marks 55, 62, 70, 48, 81, 66, 74, 59. Who is more than one std above the mean?

Solution
import pandas as pd
marks = pd.Series([55, 62, 70, 48, 81, 66, 74, 59])
z = (marks - marks.mean()) / marks.std()
print(z.round(2).tolist())
print("who is above +1:", marks[z > 1].tolist())
[-0.88, -0.22, 0.53, -1.54, 1.56, 0.15, 0.9, -0.5]
who is above +1: [81]

3. Min-max normalise the ratings 3.2, 4.5, 4.9, 2.8, 4.0.

Solution
import pandas as pd
rating = pd.Series([3.2, 4.5, 4.9, 2.8, 4.0])
mm = (rating - rating.min()) / (rating.max() - rating.min())
print(mm.round(3).tolist())
[0.19, 0.81, 1.0, 0.0, 0.571]

4. Video views: 0, 3, 12, 40, 150, 900, 25000. Reduce the skew.

Solution
import numpy as np, pandas as pd
views = pd.Series([0, 3, 12, 40, 150, 900, 25000])
print("skew before:", round(views.skew(), 2))
print("skew after :", round(np.log1p(views).skew(), 2))
skew before: 2.64
skew after : 0.68

log1p because of the 0.

Quick quiz

# Ravi:  82 in Physics  (class mean 70, std 6)
# Neha:  90 in Chemistry (class mean 84, std 4)
z = (x - mean) / std

Who did better relative to their class? A. Neha: 90 > 82 B. Ravi: z = 2.0 vs 1.5 C. Equal D. Can't compare different subjects

Answer

B. Ravi. (82 − 70) / 6 = 2.0; (90 − 84) / 4 = 1.5.

Interview questions

  1. What is a distribution? Which values a variable takes and how often.
  2. Frequency vs relative frequency? Count vs count ÷ total; use relative frequency to compare groups of different sizes.
  3. What does a bimodal distribution suggest? Often two subgroups mixed together; segment the data.
  4. Right vs left skew, and where are mean and median? Right skew: long right tail, mean > median. Left skew: long left tail, mean < median.
  5. What does kurtosis measure? What is excess kurtosis? Tail heaviness; excess kurtosis = kurtosis − 3, so a normal distribution scores 0. pandas .kurt() returns excess kurtosis.
  6. What is a z-score? Distance from the mean in standard deviations: (x − mean) / std.
  7. Standardisation vs min-max normalisation? Standardisation → mean 0, std 1, copes better with outliers; min-max → range 0–1, sensitive to outliers, good for bounded data.
  8. Why log-transform data? To reduce right skew, compress large values, and turn multiplicative effects into additive ones.
  9. How do you log-transform data containing zeros? np.log1p (log(1 + x)); invert with np.expm1.
  10. Why might scipy.stats.zscore and a pandas z-score differ? ddof 0 vs ddof 1.

Glossary

Term Meaning Example
Distribution Which values occur and how often cups per order
Frequency How many times a value occurs 37 single-cup orders
Frequency distribution / table Each value (or bin) with its frequency value_counts()
Bin A range of values grouped together 20–30 minutes
Relative frequency Frequency ÷ total; a share 0.62
Modality Number of peaks uni-, bi-, multimodal
Unimodal One peak heights
Bimodal Two peaks, often two mixed groups Mumbai orders
Tail The thin end of a distribution far from the peak very late deliveries
Skewness Number for how lopsided a distribution is s.skew() → 1.19
Right (positive) skew Long tail to the right; mean > median delivery times
Left (negative) skew Long tail to the left; mean < median easy-exam marks
Symmetric Both sides mirror each other; skew ≈ 0 heights
Kurtosis Measure of tail heaviness s.kurt()
Excess kurtosis Kurtosis − 3; 0 for a normal distribution calm −0.1, wild 27.7
Heavy (fat) tails Extreme values more common than in a bell curve "wild" deliveries
Normal distribution The symmetric bell-shaped curve (episode 42) heights
Log scale Axis where equal steps are ×10 set_yscale("log")
z-score (x − mean) / std Maths 2.5
Standardisation Converting a column to z-scores; mean 0, std 1 (s - s.mean()) / s.std()
Min-max normalisation (x − min) / (max − min); range 0–1 prices → 0…1
Logarithm The power you raise a base to; counts multiplying steps log10(1000) = 3
Log transform Applying log to every value np.log(salary)
np.log1p / np.expm1 log(1 + x) and its inverse e^x − 1 log1p(0) = 0

Summary

  • A distribution shows which values occur and how often; frequency tables count, relative frequencies compare fairly.
  • Shape has three parts: modality (how many humps), skewness (which way the tail points; mean pulled toward it) and kurtosis (how heavy the tails are).
  • A z-score puts any value on a "standard deviations from the mean" scale; standardisation gives mean 0, std 1.
  • Min-max scales to 0–1 but is ruined by outliers.
  • The log transform un-skews right-skewed data; use log1p when there are zeros.
  • Next episode: exploratory data analysis, a checklist for exploring any new dataset one variable at a time.