These are the notes for Data Analytics with Python, episode 37. Episodes 34 to 36 gave you the tools to describe one column: its centre, its spread and its shape. Today you use all of them on a brand new dataset, in a fixed order, before you trust any answer from it. That habit is exploratory data analysis (EDA). You will learn the EDA checklist, univariate analysis of numeric and categorical columns, frequency tables, distribution plots, categorical counts, rare categories, and how to turn what you see into questions and hypotheses. You will also meet ydata-profiling, a library that writes a first draft of an EDA report. Every term is defined the first time it appears and collected in the Glossary.
Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its message, as in the video.
The problem: a report that ran and was wrong
Ramesh runs a small food delivery service in Pune, Nashik and Nagpur. His app exports 600 orders. His manager asks: what is the average order value in each city? She needs it by 6 p.m. The fast way is one groupby (episode 24).
The first lines build the made-up export. Pretend you received it as a CSV file and have not looked inside yet.
import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
rng = np.random.default_rng(37); n = 600 # a made-up export
city = rng.choice(["Pune", "Nashik", "Nagpur", "pune ", "PUNE", "Goa"], n,
p=[0.47, 0.3, 0.19, 0.02, 0.015, 0.005])
km = (rng.gamma(2, 1.6, n) + 0.5).round(1)
df = pd.DataFrame({"city": city, "km": km,
"amount": rng.gamma(3, 110, n).round(),
"minutes": (15 + 4 * km + rng.gamma(2, 3, n)).round(),
"rating": rng.choice([1, 2, 3, 4, 5, np.nan], n,
p=[.04, .06, .15, .35, .25, .15])})
df.loc[3, "amount"] = 99999 # a test order
print(df.groupby("city")["amount"].mean().round())
city
Goa 507.0
Nagpur 309.0
Nashik 923.0
PUNE 355.0
Pune 335.0
pune 348.0
Name: amount, dtype: float64
No error, and the answer is nonsense: six "cities" instead of three, Pune spelled three ways (Pune, PUNE, and pune with a hidden trailing space), a Goa that Ramesh does not serve, and Nashik at 923 rupees, almost three times the others. Sent at 6 p.m., this would lead to a real decision (a premium menu for Nashik) based on garbage.
Why it happened: you analysed the data before you looked at it. Code that runs without errors can still be wrong.
What EDA is
Real-life example: a doctor's check-up. Before any diagnosis, the nurse measures temperature, pulse and weight, for every patient, every time (a checklist of routine checks). If one reading looks odd, the doctor asks why before prescribing anything. Only then does the doctor form a guess worth testing.
Exploratory data analysis (EDA) is the same three moves for data: look at the data with numbers and charts before you analyse it, to understand what is in it, catch problems, and form questions. "Exploratory" because you explore, like walking around a new city before planning routes in it. It is the first step of every serious analysis.
The EDA checklist
| Step | What you do | pandas tools |
|---|---|---|
| 1. First look | shape, column names, data types, missing values, duplicate rows | df.shape, df.columns, df.dtypes, df.isna().sum(), df.duplicated().sum(), df.info() |
| 2. Each numeric column | summary numbers and a distribution plot | describe(), nlargest/nsmallest, .skew(), histogram, box plot |
| 3. Each categorical column | counts and shares, typos, rare categories | value_counts(), value_counts(normalize=True), value_counts(dropna=False), bar chart |
| 4. Fix or note problems | write every problem and decision in the cleaning log (episode 18) | str.strip, str.title, filtering, where |
| 5. Questions and hypotheses | write down what to test next | a notebook markdown cell or a text file |
Steps 2 and 3 look at one variable at a time. A column is a variable (episode 2), so this is called univariate analysis ("uni" = one). Episode 38 is about two or more variables together (bivariate and multivariate analysis).
Step 1: the first look
The first instinct is to type the column you think exists:
print(df.shape)
print(df["City"].head())
(600, 5)
KeyError: 'City'
The above exception was the direct cause of the following exception:
KeyError: 'City'
KeyError: 'City': pandas could not find a column with that exact name. Column names are case-sensitive, so City and city are different keys. (The message appears twice because pandas re-raises the first KeyError; with a normal traceback you would see the full chain.) Never guess column names: print them.
print(df.shape)
print(df.columns.tolist())
print(pd.DataFrame({"dtype": df.dtypes, "missing": df.isna().sum()}))
print("duplicate rows:", df.duplicated().sum())
(600, 5)
['city', 'km', 'amount', 'minutes', 'rating']
dtype missing
city str 0
km float64 0
amount float64 0
minutes float64 0
rating float64 96
duplicate rows: 0
What this tells you:
- 600 rows (observations), 5 columns (variables).
cityhas dtypestr, pandas 3's text type (episode 20); the others arefloat64numbers.- Only
ratinghas missing values: 96 of them. - No exact duplicate rows.
df.info() prints the same facts in one block (non-null counts, dtypes and memory use), and is worth running too.
Step 2: numeric columns, one at a time
amount
print(df["amount"].describe().round(1))
count 600.0
mean 500.2
std 4073.7
min 14.0
25% 181.0
50% 296.0
75% 448.2
max 99999.0
Name: amount, dtype: float64
Read it like a doctor:
- The median (the 50% line) is 296 rupees, but the mean is 500. A mean far above the median means a long right tail or outliers (episode 34).
- The standard deviation is 4,074, larger than almost every order. One or more values are enormous.
- The maximum is 99,999. That is the odd pulse.
Find the rows behind it, and compare histograms with and without them:
import matplotlib.pyplot as plt
print(df.nlargest(3, "amount"))
fig, axs = plt.subplots(1, 2, figsize=(12, 4))
axs[0].hist(df["amount"], bins=40); axs[0].set_title("amount, raw")
clean = df[df["amount"] != 99999]
axs[1].hist(clean["amount"], bins=40); axs[1].set_title("amount, test order removed")
fig.tight_layout(); fig.savefig("amount.png")
print(clean["amount"].describe().round(1))
print("skew:", round(clean["amount"].skew(), 2))
city km amount minutes rating
3 Nashik 3.1 99999.0 32.0 4.0
236 Pune 2.7 1618.0 28.0 3.0
49 Nagpur 1.0 1090.0 20.0 4.0
count 599.0
mean 334.1
std 200.7
min 14.0
25% 181.0
50% 296.0
75% 448.0
max 1618.0
Name: amount, dtype: float64
skew: 1.19
Row 3 (Nashik, 99,999 rupees) is a test order someone placed while building the app; 99,999 is a classic made-up value, like 0, -1 or 9999 used as a placeholder. It alone pushed Nashik's average to 923. Without it, the median is unchanged (296: the median resists outliers), the mean drops to 334, and the skew is 1.19 (right-skewed, episode 36).

On the left, the raw histogram is a single bar: the x axis has to stretch to 100,000, so every real order falls into the first bin. On the right, without the test order, you can see the real distribution: most orders between 100 and 500 rupees, with a tail of large orders.
A distribution plot is any chart that shows how a column's values are spread out: the histogram (episode 31), the box plot (episodes 31 and 35) and the KDE curve (kernel density estimate, a smooth version of a histogram, episode 33). Look at at least one for every numeric column.
Cleaning log entry: row 3, amount 99,999, Nashik: test order, excluded from analysis; ask the app team to flag test orders.
minutes
import matplotlib.pyplot as plt
m = df["minutes"]
print(m.describe().round(1))
print("skew:", round(m.skew(), 2), " over 60 min:", (m > 60).sum())
fig, axs = plt.subplots(2, 1, figsize=(12, 5), sharex=True,
height_ratios=[3, 1])
axs[0].hist(m, bins=40); axs[0].set_title("minutes")
axs[1].boxplot(m, orientation="horizontal")
fig.tight_layout(); fig.savefig("minutes.png")
count 600.0
mean 35.2
std 9.8
min 18.0
25% 28.0
50% 34.0
75% 40.2
max 79.0
Name: minutes, dtype: float64
skew: 1.1 over 60 min: 14

The histogram shows the hump (most deliveries take 25 to 40 minutes); the box plot below, on the same x axis (sharex=True), shows the five-number summary (episode 35) and marks unusually slow deliveries as dots beyond the whisker. Skew 1.1 (right tail), 14 orders over an hour. Nothing impossible here (no negative or zero times), just a long tail worth a question.
Things to check in every numeric column: impossible values (negative ages, prices of 0, dates in the future), placeholder codes (-1, 999, 99999), a mean far from the median, a huge standard deviation, the skew, and the extremes with nlargest/nsmallest.
Step 3: categorical columns
For a category, a mean means nothing: you count. value_counts() gives each category with how often it occurs: a frequency table (episode 36).
print(df["city"].value_counts())
city
Pune 304
Nashik 172
Nagpur 101
pune 14
PUNE 8
Goa 1
Name: count, dtype: int64
Three spellings of Pune (pune has a trailing space you cannot see in the output) and a single Goa order: the whole reason the groupby showed six cities. Categorical counts catch typos faster than anything else.
Fix the spellings with the .str tools from episode 21, count as shares, and lump rare categories into "Other":
df["city"] = df["city"].str.strip().str.title()
share = df["city"].value_counts(normalize=True)
print(share.round(3))
rare = share[share < 0.01].index
df["city"] = df["city"].where(~df["city"].isin(rare), "Other")
print(df["city"].value_counts().to_dict())
city
Pune 0.543
Nashik 0.287
Nagpur 0.168
Goa 0.002
Name: proportion, dtype: float64
{'Pune': 326, 'Nashik': 172, 'Nagpur': 101, 'Other': 1}
str.strip()removes spaces at both ends;str.title()makes the first letter of each word capital.value_counts(normalize=True)gives relative frequencies (shares that add up to 1).- A rare category is a category with so few rows that you cannot learn anything reliable from it (here Goa: 1 order, 0.2%). In charts it is an invisible bar, and in models it causes trouble. The usual move is to lump all categories below a cutoff (1% here; 1-5% is common) into "Other".
s.where(condition, other)keeps values where the condition is True and replaces the rest withother.- Never silently delete what you don't understand. Log it: one Goa order: probably a test, ask the app team.
rating: an ordered category with missing values
rating is stored as a number from 1 to 5, but it is really an ordinal category (ordered categories, episode 2). Count it, and include missing values with dropna=False:
r = df["rating"]
print(r.value_counts(dropna=False).sort_index())
print("missing:", f"{r.isna().mean():.0%}")
print("mean of rated:", round(r.mean(), 2))
rating
1.0 20
2.0 20
3.0 74
4.0 227
5.0 163
NaN 96
Name: count, dtype: int64
missing: 16%
mean of rated: 3.98
Most ratings are 4 or 5 and the mean of the rated orders is 3.98, but 16% of orders have no rating. Plain value_counts() uses dropna=True and would hide those 96 rows completely.
Bar charts for categories
For categorical columns, the distribution plot is a bar chart of the counts:
import matplotlib.pyplot as plt
df["city"] = df["city"].str.strip().str.title()
fig, axs = plt.subplots(1, 2, figsize=(12, 4))
df["city"].value_counts().plot.barh(ax=axs[0], title="orders per city")
df["rating"].value_counts(dropna=False).sort_index().plot.bar(
ax=axs[1], title="rating (nan = not rated)")
fig.tight_layout(); fig.savefig("counts.png")

Pune is the biggest market; Goa's bar is barely visible; the ratings peak at 4 and the nan bar (not rated) is taller than the 1, 2 and 3 bars.
Step 5: questions and hypotheses
EDA ends by turning what you saw into what you will test.
| Seen | Question | Hypothesis |
|---|---|---|
| 14 deliveries took over 60 minutes (right tail) | Which orders are slow? | Long distances (km) make deliveries slow. |
| 16% of orders are not rated | Who skips the rating? | Slow deliveries get fewer ratings. |
| Pune has 54% of orders | Is Pune's order value different? | Pune orders are smaller because of more single-person orders. |
A hypothesis is a specific, testable guess: specific enough that data can show it right or wrong. Write hypotheses down before testing them, so you cannot quietly change the guess after seeing the answer. Episodes 46 to 49 turn this into formal hypothesis testing. Notice that every hypothesis above involves two columns: that is episode 38.
Automatic profiling reports: ydata-profiling
Doing steps 1-3 by hand for 50 columns is slow. ydata-profiling (the project started as pandas-profiling) builds one HTML report with every column's type, missing values, distribution, statistics and warnings, plus correlations and interactions between columns.
Fresh fact (checked on PyPI, 7 Nov 2026): the package has been renamed again, to fg-data-profiling (version 4.20.0, released 11 Sep 2026). The PyPI page says ydata-profiling no longer receives updates or bug fixes and asks users to migrate. Usage, from the package page:
# pip install fg-data-profiling
from data_profiling import ProfileReport # old: from ydata_profiling import ProfileReport
profile = ProfileReport(df, title="Orders profiling report")
profile.to_file("orders_report.html") # open in a browser, share with the team
(This one is not run here: it is shown exactly as documented on PyPI.) Treat the report as a first draft: it will warn that amount is skewed, but it cannot know that 99,999 is a test order or that Ramesh does not deliver in Goa. That judgement is your job.
Sources: fg-data-profiling on PyPI, ydata-profiling on PyPI.
Common mistakes
| Code / habit | Real error / symptom | Why | Fix |
|---|---|---|---|
df["City"] when the column is city |
KeyError: 'City' |
column names are case-sensitive | print(df.columns.tolist()) first |
groupby before looking |
no error; 6 cities, Nashik 923 | typos and a test order | run the EDA checklist first |
s.value_counts() on a column with gaps |
no error; missing values hidden | dropna=True by default |
value_counts(dropna=False) |
| reading only the mean | no error; outlier unnoticed | the mean is pulled by extremes | compare mean, median, max, std |
| histogram of a column with a placeholder value | one single bar | the axis stretches to the outlier | nlargest, then plot without it |
| deleting odd rows silently | no error; lost evidence | nobody can check later | write it in the cleaning log |
| treating a 1-row category as a finding | no error; noise | too few rows to trust | lump rare categories into "Other" |
Mini project: an EDA report in 15 lines
Write a function that runs steps 1-3 on any DataFrame.
import pandas as pd, numpy as np
def eda(df, rare=0.01):
print("shape:", df.shape)
print(pd.DataFrame({"dtype": df.dtypes, "missing": df.isna().sum(),
"unique": df.nunique()}))
print("duplicate rows:", df.duplicated().sum())
for col in df.select_dtypes("number"):
s = df[col]
print(f"\n{col}: median {s.median():g}, mean {s.mean():.1f}, "
f"min {s.min():g}, max {s.max():g}, skew {s.skew():.2f}")
for col in df.select_dtypes(exclude="number"):
share = df[col].value_counts(normalize=True, dropna=False)
print(f"\n{col}: {len(share)} categories; rare: {list(share[share < rare].index)}")
print(share.head(5).round(3).to_dict())
rng = np.random.default_rng(1)
shop = pd.DataFrame({"item": rng.choice(["tea", "coffee", "Tea", "juice", "lassi"], 300,
p=[0.5, 0.3, 0.1, 0.095, 0.005]),
"price": rng.choice([20, 30, 40, -1], 300, p=[0.5, 0.3, 0.17, 0.03])})
eda(shop)
shape: (300, 2)
dtype missing unique
item str 0 5
price int64 0 4
duplicate rows: 285
price: median 20, mean 25.7, min -1, max 40, skew -0.10
item: 5 categories; rare: ['lassi']
{'tea': 0.51, 'coffee': 0.303, 'Tea': 0.093, 'juice': 0.087, 'lassi': 0.007}
The report flags the planted problems: a Tea/tea spelling split, a rare lassi, and a min of -1 (a placeholder for "unknown price"). It also reports 285 duplicate rows: with only two columns and a handful of possible values, many orders are identical, which is normal here. That is why real order data needs an ID column (like order_id) before duplicates mean anything.
Practice
1. A column of customer ages: [25, 31, 0, 45, 38, 29, 999, 33, 41, 27]. Run describe(). Which values are impossible, and what happens to the mean if you mark them as missing?
Solution
import numpy as np, pandas as pd
age = pd.Series([25, 31, 0, 45, 38, 29, 999, 33, 41, 27])
print(age.describe().round(1))
clean = age.where(age.between(1, 120)) # outside 1-120 -> NaN
print("mean before:", age.mean(), " after:", round(clean.mean(), 1), " missing:", clean.isna().sum())
count 10.0
mean 126.8
std 306.7
min 0.0
25% 27.5
50% 32.0
75% 40.2
max 999.0
dtype: float64
mean before: 126.8 after: 33.6 missing: 2
0 and 999 are impossible ages (placeholders). Marking them missing changes the mean from 126.8 to 33.6, a believable value.
2. City names ["Delhi", "delhi", " Delhi", "Mumbai", "MUMBAI", "Pune"]: count them, then clean and count again.
Solution
import pandas as pd
c = pd.Series(["Delhi", "delhi", " Delhi", "Mumbai", "MUMBAI", "Pune"])
print(c.value_counts().to_dict())
print(c.str.strip().str.title().value_counts().to_dict())
{'Delhi': 1, 'delhi': 1, ' Delhi': 1, 'Mumbai': 1, 'MUMBAI': 1, 'Pune': 1}
{'Delhi': 3, 'Mumbai': 2, 'Pune': 1}
3. Payment methods: UPI 520, card 300, cash 170, cheque 6, crypto 4. Lump categories under 2% into "Other".
Solution
import pandas as pd
pay = pd.Series(["UPI"] * 520 + ["card"] * 300 + ["cash"] * 170 + ["cheque"] * 6 + ["crypto"] * 4)
share = pay.value_counts(normalize=True)
rare = share[share < 0.02].index
print(pay.where(~pay.isin(rare), "Other").value_counts().to_dict())
{'UPI': 520, 'card': 300, 'cash': 170, 'Other': 10}
4. Write two questions and two hypotheses for a dataset of exam marks with columns student, hours_studied, marks, school.
Solution
Questions: Do students who study more score more? Do schools differ in average marks? Hypotheses: "Marks rise with hours studied." "School A's median mark is higher than School B's." Each is specific and testable with the data, and needs two columns (episode 38).
Quick quiz
import numpy as np, pandas as pd
rng = np.random.default_rng(7)
age = pd.Series(rng.normal(33, 9, 500).round().clip(18, 70))
age[rng.choice(500, 6, replace=False)] = [-1, -1, -1, 999, 999, 999]
print(age.describe().round(1))
count 500.0
mean 37.6
std 75.2
min -1.0
25% 26.0
50% 32.0
75% 38.0
max 999.0
dtype: float64
What should you notice first? A. Ages are normally distributed B. Impossible values: -1 and 999 C. The mean age is 37.6, report it D. There are 500 missing ages
Answer
B. -1 and 999 are impossible ages: codes for "unknown". They also inflate the standard deviation (75.2) and pull the mean (37.6) above the median (32). Find those rows, ask what they mean, and mark them as missing.
Interview questions
- What is EDA and why do it first? Exploring data with summaries and charts before analysis, to understand it, find problems, and form hypotheses. Code can run without errors and still give wrong answers on dirty data.
- What are the steps of your EDA checklist? First look (shape, columns, dtypes, missing, duplicates); each numeric column (describe, histogram/box plot, extremes); each categorical column (value_counts, typos, rare categories); log fixes; write hypotheses.
- What is univariate analysis? Analysing one variable at a time: its centre, spread, shape and counts.
- How do you spot outliers or placeholder values in a numeric column? Compare mean vs median, look at std, min, max,
nlargest/nsmallest, histograms and box plots, and check domain limits. - What is a rare category and how do you handle it? A category with too few rows to be reliable; group rare categories into "Other" and investigate why they exist.
- Why
value_counts(dropna=False)? The default drops missing values, hiding how much is missing. - What is a hypothesis in EDA? A specific, testable guess written before testing, e.g. "slow deliveries get fewer ratings".
- What do automated profiling tools do, and what are their limits? Libraries like ydata-profiling (now fg-data-profiling) build a full HTML summary of every column; they cannot apply domain knowledge, like knowing a value is a test order.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Exploratory data analysis (EDA) | Looking at data with numbers and charts before analysing it, to understand it and catch problems | today's checklist |
| EDA checklist | A fixed list of checks run on every new dataset | first look → numeric → categorical → log → hypotheses |
| Variable | One column; something measured for every row | minutes |
| Univariate analysis | Analysing one variable at a time | describe() of amount |
| Bivariate / multivariate analysis | Two / three or more variables together (episode 38) | km vs minutes |
KeyError |
Error raised when a key (here a column name) is not found | df["City"] |
| Case-sensitive | Capital and small letters count as different | City ≠ city |
| Data type (dtype) | What kind of values a column holds | float64, str |
| Frequency table | Each value with how often it occurs | value_counts() |
| Relative frequency | A count divided by the total; a share | Pune 0.543 |
| Distribution plot | A chart of how a column's values are spread | histogram, box plot, KDE |
| Histogram | Bars showing how many values fall in each range (bin) | minutes histogram |
| Box plot | Box from Q1 to Q3, line at the median, whiskers, dots for outliers | minutes box plot |
| KDE | Kernel density estimate: a smooth curve version of a histogram | sns.kdeplot |
| Categorical counts | Counting how often each category occurs | orders per city |
| Rare category | A category with too few rows to be reliable | Goa, 1 order |
| "Other" bucket | One category that collects all rare categories | where(~isin(rare), "Other") |
| Ordinal | Categories with a natural order | ratings 1-5 |
| Placeholder value | A fake value meaning "unknown" or "test" | 99999, -1, 999 |
| Test order | An order placed while building or testing a system, not by a real customer | row 3 |
| Cleaning log | A written record of every problem found and decision made (episode 18) | "row 3 excluded" |
| Hypothesis | A specific, testable guess, written before testing | "slow deliveries get fewer ratings" |
| Profiling report | An automatic summary of every column of a dataset | ProfileReport(df) |
| ydata-profiling / fg-data-profiling | Python library that builds an HTML EDA report; renamed to fg-data-profiling in 2026 | profile.to_file("r.html") |
Summary
- No error does not mean correct. Analysing before looking gave six cities and a fake Nashik average.
- EDA = look before you analyse. Run the checklist: first look, numeric columns, categorical columns, log every problem, write hypotheses.
- Univariate analysis:
describe(),nlargest, skew, histograms and box plots for numbers;value_counts()(withnormalize=Trueanddropna=False) and bar charts for categories. - Watch for placeholder values, typos, rare categories (lump into "Other") and hidden missing values.
- End with questions and hypotheses written down before testing.
- ydata-profiling (now fg-data-profiling) gives an automatic first draft; your judgement does the rest.
- Next episode: EDA 2: relationships between variables: testing these hypotheses with scatter plots, grouped box plots and crosstabs.