SyntaxPilot
Courses/Data Analytics with Python/Lesson 37
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Data Analytics with Python · Lesson 37 of 90

EDA 1: exploring one variable at a time

exploratory data analysis (EDA)EDA checklistunivariate analysisfrequency tabledistribution plotscategorical countsrare categoriesydata-profiling (mention)forming questions and hypotheses

These are the notes for Data Analytics with Python, episode 37. Episodes 34 to 36 gave you the tools to describe one column: its centre, its spread and its shape. Today you use all of them on a brand new dataset, in a fixed order, before you trust any answer from it. That habit is exploratory data analysis (EDA). You will learn the EDA checklist, univariate analysis of numeric and categorical columns, frequency tables, distribution plots, categorical counts, rare categories, and how to turn what you see into questions and hypotheses. You will also meet ydata-profiling, a library that writes a first draft of an EDA report. Every term is defined the first time it appears and collected in the Glossary.

Every program below was run for real (Python 3.13, NumPy 2.5.3, pandas 3.0.5, matplotlib 3.11.2, on a server with no screen). Outputs and images are from those runs. All data is made up for teaching. Programs start with sys.tracebacklimit = 0 so an error shows only its message, as in the video.

The problem: a report that ran and was wrong

Ramesh runs a small food delivery service in Pune, Nashik and Nagpur. His app exports 600 orders. His manager asks: what is the average order value in each city? She needs it by 6 p.m. The fast way is one groupby (episode 24).

The first lines build the made-up export. Pretend you received it as a CSV file and have not looked inside yet.

import sys; sys.tracebacklimit = 0
import numpy as np, pandas as pd
rng = np.random.default_rng(37); n = 600            # a made-up export
city = rng.choice(["Pune", "Nashik", "Nagpur", "pune ", "PUNE", "Goa"], n,
                  p=[0.47, 0.3, 0.19, 0.02, 0.015, 0.005])
km = (rng.gamma(2, 1.6, n) + 0.5).round(1)
df = pd.DataFrame({"city": city, "km": km,
    "amount": rng.gamma(3, 110, n).round(),
    "minutes": (15 + 4 * km + rng.gamma(2, 3, n)).round(),
    "rating": rng.choice([1, 2, 3, 4, 5, np.nan], n,
                         p=[.04, .06, .15, .35, .25, .15])})
df.loc[3, "amount"] = 99999                         # a test order
print(df.groupby("city")["amount"].mean().round())
city
Goa       507.0
Nagpur    309.0
Nashik    923.0
PUNE      355.0
Pune      335.0
pune      348.0
Name: amount, dtype: float64

No error, and the answer is nonsense: six "cities" instead of three, Pune spelled three ways (Pune, PUNE, and pune with a hidden trailing space), a Goa that Ramesh does not serve, and Nashik at 923 rupees, almost three times the others. Sent at 6 p.m., this would lead to a real decision (a premium menu for Nashik) based on garbage.

Why it happened: you analysed the data before you looked at it. Code that runs without errors can still be wrong.

What EDA is

Real-life example: a doctor's check-up. Before any diagnosis, the nurse measures temperature, pulse and weight, for every patient, every time (a checklist of routine checks). If one reading looks odd, the doctor asks why before prescribing anything. Only then does the doctor form a guess worth testing.

Exploratory data analysis (EDA) is the same three moves for data: look at the data with numbers and charts before you analyse it, to understand what is in it, catch problems, and form questions. "Exploratory" because you explore, like walking around a new city before planning routes in it. It is the first step of every serious analysis.

The EDA checklist

Step What you do pandas tools
1. First look shape, column names, data types, missing values, duplicate rows df.shape, df.columns, df.dtypes, df.isna().sum(), df.duplicated().sum(), df.info()
2. Each numeric column summary numbers and a distribution plot describe(), nlargest/nsmallest, .skew(), histogram, box plot
3. Each categorical column counts and shares, typos, rare categories value_counts(), value_counts(normalize=True), value_counts(dropna=False), bar chart
4. Fix or note problems write every problem and decision in the cleaning log (episode 18) str.strip, str.title, filtering, where
5. Questions and hypotheses write down what to test next a notebook markdown cell or a text file

Steps 2 and 3 look at one variable at a time. A column is a variable (episode 2), so this is called univariate analysis ("uni" = one). Episode 38 is about two or more variables together (bivariate and multivariate analysis).

Step 1: the first look

The first instinct is to type the column you think exists:

print(df.shape)
print(df["City"].head())
(600, 5)
KeyError: 'City'

The above exception was the direct cause of the following exception:

KeyError: 'City'

KeyError: 'City': pandas could not find a column with that exact name. Column names are case-sensitive, so City and city are different keys. (The message appears twice because pandas re-raises the first KeyError; with a normal traceback you would see the full chain.) Never guess column names: print them.

print(df.shape)
print(df.columns.tolist())
print(pd.DataFrame({"dtype": df.dtypes, "missing": df.isna().sum()}))
print("duplicate rows:", df.duplicated().sum())
(600, 5)
['city', 'km', 'amount', 'minutes', 'rating']
           dtype  missing
city         str        0
km       float64        0
amount   float64        0
minutes  float64        0
rating   float64       96
duplicate rows: 0

What this tells you:

  • 600 rows (observations), 5 columns (variables).
  • city has dtype str, pandas 3's text type (episode 20); the others are float64 numbers.
  • Only rating has missing values: 96 of them.
  • No exact duplicate rows.

df.info() prints the same facts in one block (non-null counts, dtypes and memory use), and is worth running too.

Step 2: numeric columns, one at a time

amount

print(df["amount"].describe().round(1))
count      600.0
mean       500.2
std       4073.7
min         14.0
25%        181.0
50%        296.0
75%        448.2
max      99999.0
Name: amount, dtype: float64

Read it like a doctor:

  • The median (the 50% line) is 296 rupees, but the mean is 500. A mean far above the median means a long right tail or outliers (episode 34).
  • The standard deviation is 4,074, larger than almost every order. One or more values are enormous.
  • The maximum is 99,999. That is the odd pulse.

Find the rows behind it, and compare histograms with and without them:

import matplotlib.pyplot as plt
print(df.nlargest(3, "amount"))
fig, axs = plt.subplots(1, 2, figsize=(12, 4))
axs[0].hist(df["amount"], bins=40); axs[0].set_title("amount, raw")
clean = df[df["amount"] != 99999]
axs[1].hist(clean["amount"], bins=40); axs[1].set_title("amount, test order removed")
fig.tight_layout(); fig.savefig("amount.png")
print(clean["amount"].describe().round(1))
print("skew:", round(clean["amount"].skew(), 2))
       city   km   amount  minutes  rating
3    Nashik  3.1  99999.0     32.0     4.0
236    Pune  2.7   1618.0     28.0     3.0
49   Nagpur  1.0   1090.0     20.0     4.0
count     599.0
mean      334.1
std       200.7
min        14.0
25%       181.0
50%       296.0
75%       448.0
max      1618.0
Name: amount, dtype: float64
skew: 1.19

Row 3 (Nashik, 99,999 rupees) is a test order someone placed while building the app; 99,999 is a classic made-up value, like 0, -1 or 9999 used as a placeholder. It alone pushed Nashik's average to 923. Without it, the median is unchanged (296: the median resists outliers), the mean drops to 334, and the skew is 1.19 (right-skewed, episode 36).

amount histograms

On the left, the raw histogram is a single bar: the x axis has to stretch to 100,000, so every real order falls into the first bin. On the right, without the test order, you can see the real distribution: most orders between 100 and 500 rupees, with a tail of large orders.

A distribution plot is any chart that shows how a column's values are spread out: the histogram (episode 31), the box plot (episodes 31 and 35) and the KDE curve (kernel density estimate, a smooth version of a histogram, episode 33). Look at at least one for every numeric column.

Cleaning log entry: row 3, amount 99,999, Nashik: test order, excluded from analysis; ask the app team to flag test orders.

minutes

import matplotlib.pyplot as plt
m = df["minutes"]
print(m.describe().round(1))
print("skew:", round(m.skew(), 2), " over 60 min:", (m > 60).sum())
fig, axs = plt.subplots(2, 1, figsize=(12, 5), sharex=True,
                        height_ratios=[3, 1])
axs[0].hist(m, bins=40); axs[0].set_title("minutes")
axs[1].boxplot(m, orientation="horizontal")
fig.tight_layout(); fig.savefig("minutes.png")
count    600.0
mean      35.2
std        9.8
min       18.0
25%       28.0
50%       34.0
75%       40.2
max       79.0
Name: minutes, dtype: float64
skew: 1.1  over 60 min: 14

minutes histogram and box plot

The histogram shows the hump (most deliveries take 25 to 40 minutes); the box plot below, on the same x axis (sharex=True), shows the five-number summary (episode 35) and marks unusually slow deliveries as dots beyond the whisker. Skew 1.1 (right tail), 14 orders over an hour. Nothing impossible here (no negative or zero times), just a long tail worth a question.

Things to check in every numeric column: impossible values (negative ages, prices of 0, dates in the future), placeholder codes (-1, 999, 99999), a mean far from the median, a huge standard deviation, the skew, and the extremes with nlargest/nsmallest.

Step 3: categorical columns

For a category, a mean means nothing: you count. value_counts() gives each category with how often it occurs: a frequency table (episode 36).

print(df["city"].value_counts())
city
Pune      304
Nashik    172
Nagpur    101
pune       14
PUNE        8
Goa         1
Name: count, dtype: int64

Three spellings of Pune (pune has a trailing space you cannot see in the output) and a single Goa order: the whole reason the groupby showed six cities. Categorical counts catch typos faster than anything else.

Fix the spellings with the .str tools from episode 21, count as shares, and lump rare categories into "Other":

df["city"] = df["city"].str.strip().str.title()
share = df["city"].value_counts(normalize=True)
print(share.round(3))
rare = share[share < 0.01].index
df["city"] = df["city"].where(~df["city"].isin(rare), "Other")
print(df["city"].value_counts().to_dict())
city
Pune      0.543
Nashik    0.287
Nagpur    0.168
Goa       0.002
Name: proportion, dtype: float64
{'Pune': 326, 'Nashik': 172, 'Nagpur': 101, 'Other': 1}
  • str.strip() removes spaces at both ends; str.title() makes the first letter of each word capital.
  • value_counts(normalize=True) gives relative frequencies (shares that add up to 1).
  • A rare category is a category with so few rows that you cannot learn anything reliable from it (here Goa: 1 order, 0.2%). In charts it is an invisible bar, and in models it causes trouble. The usual move is to lump all categories below a cutoff (1% here; 1-5% is common) into "Other".
  • s.where(condition, other) keeps values where the condition is True and replaces the rest with other.
  • Never silently delete what you don't understand. Log it: one Goa order: probably a test, ask the app team.

rating: an ordered category with missing values

rating is stored as a number from 1 to 5, but it is really an ordinal category (ordered categories, episode 2). Count it, and include missing values with dropna=False:

r = df["rating"]
print(r.value_counts(dropna=False).sort_index())
print("missing:", f"{r.isna().mean():.0%}")
print("mean of rated:", round(r.mean(), 2))
rating
1.0     20
2.0     20
3.0     74
4.0    227
5.0    163
NaN     96
Name: count, dtype: int64
missing: 16%
mean of rated: 3.98

Most ratings are 4 or 5 and the mean of the rated orders is 3.98, but 16% of orders have no rating. Plain value_counts() uses dropna=True and would hide those 96 rows completely.

Bar charts for categories

For categorical columns, the distribution plot is a bar chart of the counts:

import matplotlib.pyplot as plt
df["city"] = df["city"].str.strip().str.title()
fig, axs = plt.subplots(1, 2, figsize=(12, 4))
df["city"].value_counts().plot.barh(ax=axs[0], title="orders per city")
df["rating"].value_counts(dropna=False).sort_index().plot.bar(
    ax=axs[1], title="rating (nan = not rated)")
fig.tight_layout(); fig.savefig("counts.png")

bar charts

Pune is the biggest market; Goa's bar is barely visible; the ratings peak at 4 and the nan bar (not rated) is taller than the 1, 2 and 3 bars.

Step 5: questions and hypotheses

EDA ends by turning what you saw into what you will test.

Seen Question Hypothesis
14 deliveries took over 60 minutes (right tail) Which orders are slow? Long distances (km) make deliveries slow.
16% of orders are not rated Who skips the rating? Slow deliveries get fewer ratings.
Pune has 54% of orders Is Pune's order value different? Pune orders are smaller because of more single-person orders.

A hypothesis is a specific, testable guess: specific enough that data can show it right or wrong. Write hypotheses down before testing them, so you cannot quietly change the guess after seeing the answer. Episodes 46 to 49 turn this into formal hypothesis testing. Notice that every hypothesis above involves two columns: that is episode 38.

Automatic profiling reports: ydata-profiling

Doing steps 1-3 by hand for 50 columns is slow. ydata-profiling (the project started as pandas-profiling) builds one HTML report with every column's type, missing values, distribution, statistics and warnings, plus correlations and interactions between columns.

Fresh fact (checked on PyPI, 7 Nov 2026): the package has been renamed again, to fg-data-profiling (version 4.20.0, released 11 Sep 2026). The PyPI page says ydata-profiling no longer receives updates or bug fixes and asks users to migrate. Usage, from the package page:

# pip install fg-data-profiling
from data_profiling import ProfileReport        # old: from ydata_profiling import ProfileReport

profile = ProfileReport(df, title="Orders profiling report")
profile.to_file("orders_report.html")            # open in a browser, share with the team

(This one is not run here: it is shown exactly as documented on PyPI.) Treat the report as a first draft: it will warn that amount is skewed, but it cannot know that 99,999 is a test order or that Ramesh does not deliver in Goa. That judgement is your job.

Sources: fg-data-profiling on PyPI, ydata-profiling on PyPI.

Common mistakes

Code / habit Real error / symptom Why Fix
df["City"] when the column is city KeyError: 'City' column names are case-sensitive print(df.columns.tolist()) first
groupby before looking no error; 6 cities, Nashik 923 typos and a test order run the EDA checklist first
s.value_counts() on a column with gaps no error; missing values hidden dropna=True by default value_counts(dropna=False)
reading only the mean no error; outlier unnoticed the mean is pulled by extremes compare mean, median, max, std
histogram of a column with a placeholder value one single bar the axis stretches to the outlier nlargest, then plot without it
deleting odd rows silently no error; lost evidence nobody can check later write it in the cleaning log
treating a 1-row category as a finding no error; noise too few rows to trust lump rare categories into "Other"

Mini project: an EDA report in 15 lines

Write a function that runs steps 1-3 on any DataFrame.

import pandas as pd, numpy as np

def eda(df, rare=0.01):
    print("shape:", df.shape)
    print(pd.DataFrame({"dtype": df.dtypes, "missing": df.isna().sum(),
                        "unique": df.nunique()}))
    print("duplicate rows:", df.duplicated().sum())
    for col in df.select_dtypes("number"):
        s = df[col]
        print(f"\n{col}: median {s.median():g}, mean {s.mean():.1f}, "
              f"min {s.min():g}, max {s.max():g}, skew {s.skew():.2f}")
    for col in df.select_dtypes(exclude="number"):
        share = df[col].value_counts(normalize=True, dropna=False)
        print(f"\n{col}: {len(share)} categories; rare: {list(share[share < rare].index)}")
        print(share.head(5).round(3).to_dict())

rng = np.random.default_rng(1)
shop = pd.DataFrame({"item": rng.choice(["tea", "coffee", "Tea", "juice", "lassi"], 300,
                                         p=[0.5, 0.3, 0.1, 0.095, 0.005]),
                     "price": rng.choice([20, 30, 40, -1], 300, p=[0.5, 0.3, 0.17, 0.03])})
eda(shop)
shape: (300, 2)
       dtype  missing  unique
item     str        0       5
price  int64        0       4
duplicate rows: 285

price: median 20, mean 25.7, min -1, max 40, skew -0.10

item: 5 categories; rare: ['lassi']
{'tea': 0.51, 'coffee': 0.303, 'Tea': 0.093, 'juice': 0.087, 'lassi': 0.007}

The report flags the planted problems: a Tea/tea spelling split, a rare lassi, and a min of -1 (a placeholder for "unknown price"). It also reports 285 duplicate rows: with only two columns and a handful of possible values, many orders are identical, which is normal here. That is why real order data needs an ID column (like order_id) before duplicates mean anything.

Practice

1. A column of customer ages: [25, 31, 0, 45, 38, 29, 999, 33, 41, 27]. Run describe(). Which values are impossible, and what happens to the mean if you mark them as missing?

Solution
import numpy as np, pandas as pd
age = pd.Series([25, 31, 0, 45, 38, 29, 999, 33, 41, 27])
print(age.describe().round(1))
clean = age.where(age.between(1, 120))       # outside 1-120 -> NaN
print("mean before:", age.mean(), " after:", round(clean.mean(), 1), " missing:", clean.isna().sum())
count     10.0
mean     126.8
std      306.7
min        0.0
25%       27.5
50%       32.0
75%       40.2
max      999.0
dtype: float64
mean before: 126.8  after: 33.6  missing: 2

0 and 999 are impossible ages (placeholders). Marking them missing changes the mean from 126.8 to 33.6, a believable value.

2. City names ["Delhi", "delhi", " Delhi", "Mumbai", "MUMBAI", "Pune"]: count them, then clean and count again.

Solution
import pandas as pd
c = pd.Series(["Delhi", "delhi", " Delhi", "Mumbai", "MUMBAI", "Pune"])
print(c.value_counts().to_dict())
print(c.str.strip().str.title().value_counts().to_dict())
{'Delhi': 1, 'delhi': 1, ' Delhi': 1, 'Mumbai': 1, 'MUMBAI': 1, 'Pune': 1}
{'Delhi': 3, 'Mumbai': 2, 'Pune': 1}

3. Payment methods: UPI 520, card 300, cash 170, cheque 6, crypto 4. Lump categories under 2% into "Other".

Solution
import pandas as pd
pay = pd.Series(["UPI"] * 520 + ["card"] * 300 + ["cash"] * 170 + ["cheque"] * 6 + ["crypto"] * 4)
share = pay.value_counts(normalize=True)
rare = share[share < 0.02].index
print(pay.where(~pay.isin(rare), "Other").value_counts().to_dict())
{'UPI': 520, 'card': 300, 'cash': 170, 'Other': 10}

4. Write two questions and two hypotheses for a dataset of exam marks with columns student, hours_studied, marks, school.

Solution

Questions: Do students who study more score more? Do schools differ in average marks? Hypotheses: "Marks rise with hours studied." "School A's median mark is higher than School B's." Each is specific and testable with the data, and needs two columns (episode 38).

Quick quiz

import numpy as np, pandas as pd
rng = np.random.default_rng(7)
age = pd.Series(rng.normal(33, 9, 500).round().clip(18, 70))
age[rng.choice(500, 6, replace=False)] = [-1, -1, -1, 999, 999, 999]
print(age.describe().round(1))
count    500.0
mean      37.6
std       75.2
min       -1.0
25%       26.0
50%       32.0
75%       38.0
max      999.0
dtype: float64

What should you notice first? A. Ages are normally distributed B. Impossible values: -1 and 999 C. The mean age is 37.6, report it D. There are 500 missing ages

Answer

B. -1 and 999 are impossible ages: codes for "unknown". They also inflate the standard deviation (75.2) and pull the mean (37.6) above the median (32). Find those rows, ask what they mean, and mark them as missing.

Interview questions

  1. What is EDA and why do it first? Exploring data with summaries and charts before analysis, to understand it, find problems, and form hypotheses. Code can run without errors and still give wrong answers on dirty data.
  2. What are the steps of your EDA checklist? First look (shape, columns, dtypes, missing, duplicates); each numeric column (describe, histogram/box plot, extremes); each categorical column (value_counts, typos, rare categories); log fixes; write hypotheses.
  3. What is univariate analysis? Analysing one variable at a time: its centre, spread, shape and counts.
  4. How do you spot outliers or placeholder values in a numeric column? Compare mean vs median, look at std, min, max, nlargest/nsmallest, histograms and box plots, and check domain limits.
  5. What is a rare category and how do you handle it? A category with too few rows to be reliable; group rare categories into "Other" and investigate why they exist.
  6. Why value_counts(dropna=False)? The default drops missing values, hiding how much is missing.
  7. What is a hypothesis in EDA? A specific, testable guess written before testing, e.g. "slow deliveries get fewer ratings".
  8. What do automated profiling tools do, and what are their limits? Libraries like ydata-profiling (now fg-data-profiling) build a full HTML summary of every column; they cannot apply domain knowledge, like knowing a value is a test order.

Glossary

Term Meaning Example
Exploratory data analysis (EDA) Looking at data with numbers and charts before analysing it, to understand it and catch problems today's checklist
EDA checklist A fixed list of checks run on every new dataset first look → numeric → categorical → log → hypotheses
Variable One column; something measured for every row minutes
Univariate analysis Analysing one variable at a time describe() of amount
Bivariate / multivariate analysis Two / three or more variables together (episode 38) km vs minutes
KeyError Error raised when a key (here a column name) is not found df["City"]
Case-sensitive Capital and small letters count as different City ≠ city
Data type (dtype) What kind of values a column holds float64, str
Frequency table Each value with how often it occurs value_counts()
Relative frequency A count divided by the total; a share Pune 0.543
Distribution plot A chart of how a column's values are spread histogram, box plot, KDE
Histogram Bars showing how many values fall in each range (bin) minutes histogram
Box plot Box from Q1 to Q3, line at the median, whiskers, dots for outliers minutes box plot
KDE Kernel density estimate: a smooth curve version of a histogram sns.kdeplot
Categorical counts Counting how often each category occurs orders per city
Rare category A category with too few rows to be reliable Goa, 1 order
"Other" bucket One category that collects all rare categories where(~isin(rare), "Other")
Ordinal Categories with a natural order ratings 1-5
Placeholder value A fake value meaning "unknown" or "test" 99999, -1, 999
Test order An order placed while building or testing a system, not by a real customer row 3
Cleaning log A written record of every problem found and decision made (episode 18) "row 3 excluded"
Hypothesis A specific, testable guess, written before testing "slow deliveries get fewer ratings"
Profiling report An automatic summary of every column of a dataset ProfileReport(df)
ydata-profiling / fg-data-profiling Python library that builds an HTML EDA report; renamed to fg-data-profiling in 2026 profile.to_file("r.html")

Summary

  • No error does not mean correct. Analysing before looking gave six cities and a fake Nashik average.
  • EDA = look before you analyse. Run the checklist: first look, numeric columns, categorical columns, log every problem, write hypotheses.
  • Univariate analysis: describe(), nlargest, skew, histograms and box plots for numbers; value_counts() (with normalize=True and dropna=False) and bar charts for categories.
  • Watch for placeholder values, typos, rare categories (lump into "Other") and hidden missing values.
  • End with questions and hypotheses written down before testing.
  • ydata-profiling (now fg-data-profiling) gives an automatic first draft; your judgement does the rest.
  • Next episode: EDA 2: relationships between variables: testing these hypotheses with scatter plots, grouped box plots and crosstabs.