These are the notes for Data Analytics with Python, episode 2. You will learn what a dataset is (observations and variables), the types of data (quantitative vs qualitative, discrete vs continuous, nominal vs ordinal), the four levels of measurement, tidy data, structured vs semi-structured vs unstructured data, primary vs secondary data, and the five file formats analysts meet every week. Every term is defined the first time it appears and collected in the Glossary. Every program runs as-is in the SyntaxPilot compiler, and every output is the real output from Python 3.13 (pandas and pyarrow installed).
The problem: marks stuck in a sentence
Mrs Iyer, a class teacher with 40 students and 6 subjects, writes marks in her diary as sentences: "Riya got 82 in maths, Aman got 75, Sara got 91". When the principal asks for the class average, she has to read every sentence again. Python can't do better with a sentence:
notes = "Riya got 82 in maths, Aman got 75, Sara got 91"
print(notes)
print(sum(notes))
Output:
Riya got 82 in maths, Aman got 75, Sara got 91
Traceback (most recent call last):
...
TypeError: unsupported operand type(s) for +: 'int' and 'str'
Why: a sentence has no structure. To Python it's just characters; nothing says which part is a name and which part is a mark.
Fix: a table, like a school register: one row per student, one column per fact.
Datasets: observations and variables
| School register | Data word | Meaning |
|---|---|---|
| The whole register | Dataset | A collection of data about one topic, usually a table |
| One line (everything about Riya) | Observation = record = row | Everything recorded about one thing |
| One heading (Marks) | Variable = feature = column | One fact recorded for every row. It "varies" from row to row. Feature is the machine-learning word |
In Python, each student can be a dictionary: values with labels ("marks": 82).
# each row = one student, each column = one variable
students = [
{"name": "Riya", "marks": 82, "size": "M", "pin": 400053},
{"name": "Aman", "marks": 75, "size": "L", "pin": 110001},
{"name": "Sara", "marks": 91, "size": "S", "pin": 560001},
]
marks = [s["marks"] for s in students]
print("Average marks:", sum(marks) / len(marks))
pins = [s["pin"] for s in students]
print("Average pin code:", sum(pins) / len(pins))
Output:
Average marks: 82.66666666666667
Average pin code: 356685.0
The average of marks is meaningful. The average of pin codes ran without an error and means nothing: 356685 isn't a place. And the average of sizes crashes:
sizes = ["M", "L", "S"]
print(sum(sizes) / len(sizes))
TypeError: unsupported operand type(s) for +: 'int' and 'str'
Rule 1 of analytics: before you calculate anything, ask what type of data the column holds. No error does not mean a correct answer.
Types of data
Data
├── Quantitative (numerical): an amount you can do maths on
│ ├── Discrete: counted, whole steps → cups sold, number of children
│ └── Continuous: measured, any value → weight 61.37 kg, time 9.58 s
└── Qualitative (categorical): a label / group
├── Nominal: no order → blood group, city, pin code
└── Ordinal: a natural order → S < M < L, 1–5 star rating
| Type | Plain meaning | Examples | Sensible maths |
|---|---|---|---|
| Quantitative (numerical) | An amount | marks, price, height | sum, average, … |
| Discrete | Counted; jumps in whole steps | cups sold, students present | counts, averages |
| Continuous | Measured; any value in between | weight, time, temperature | averages, ranges |
| Qualitative (categorical) | A label that puts a row in a group | city, size, blood group | counting per group |
| Nominal | Categories with no order | blood group A/B/O, pin code | counts, the most common one |
| Ordinal | Categories with a natural order | S/M/L, star ratings, class rank | order, median |
Digits don't make a number: pin codes, phone numbers and roll numbers are categorical (nominal). Count them, don't average them:
pins = ["400053", "110001", "400053", "560001"]
counts = {}
for p in pins:
counts[p] = counts.get(p, 0) + 1
print(counts)
{'400053': 2, '110001': 1, '560001': 1}
The ordinal trap: sorting sizes
sizes = ["M", "L", "S", "M"]
print(sorted(sizes))
order = {"S": 1, "M": 2, "L": 3}
print(sorted(sizes, key=order.get))
['L', 'M', 'M', 'S']
['S', 'M', 'M', 'L']
Python sorts letters alphabetically; it doesn't know small comes before large. Tell it the order with a dictionary that maps each size to a rank. (pandas has an ordered categorical type for this; we'll use it later in the course.)
The four levels of measurement
Each level allows more maths than the one before.
| Level | Plain meaning | Example | Allowed |
|---|---|---|---|
| Nominal | Names only, no order | Blood group, city | = and ≠, counting |
| Ordinal | An order, but gaps aren't equal | Hotel stars, S/M/L | also < and >, median |
| Interval | Equal gaps, but no true zero | Temperature in °C, calendar year | also + and − (differences) |
| Ratio | Equal gaps and a true zero (0 = none) | Money, weight, cups, age | everything, including "twice as much" |
Proof that "40 °C is twice as hot as 20 °C" is false:
def to_f(c):
return c * 9 / 5 + 32
print("Celsius:", 40 / 20)
print("Fahrenheit:", to_f(40) / to_f(20))
print("Rupees:", 40 / 20)
Celsius: 2.0
Fahrenheit: 1.5294117647058822
Rupees: 2.0
The ratio changes when only the unit changes (2 vs 1.53), so it was never a real fact. 0 °C doesn't mean "no heat". ₹40 really is twice ₹20, because ₹0 really means no money.
Tidy data
Same numbers, two layouts:
Untidy (months as columns):
stall,Jan,Feb,Mar
Andheri,9100,8800,9400
Bandra,8200,7900,8600
Tidy (one row per observation):
stall,month,cups
Andheri,Jan,9100
Andheri,Feb,8800
Andheri,Mar,9400
Bandra,Jan,8200
Bandra,Feb,7900
Bandra,Mar,8600
Tidy data rules: every column is one variable, every row is one observation, every cell holds one value. In the untidy table the variable month hides in the headers: adding April means adding a column, and "best month" means checking every column. Python and pandas tools expect tidy tables (later we'll switch shapes with one line of pandas: melt and pivot).
Structured, semi-structured and unstructured data
| Kind | Plain meaning | Everyday example | Typical files |
|---|---|---|---|
| Structured | Fixed rows and columns, same fields for every row | Bank statement: date, amount, balance | CSV, Excel, database tables |
| Semi-structured | Labels and nesting, but fields can differ per record | A phone contact: two numbers for one person, email for another | JSON, XML |
| Unstructured | No fields at all | Photos, voice notes, written reviews | images, audio, free text |
Unstructured data has to be processed (for example, reading the stars and keywords out of reviews) before an analyst can count it.
Primary vs secondary data
| Primary data | Secondary data | |
|---|---|---|
| Who collected it | You, for your own question | Someone else, for another purpose |
| Kitchen analogy | Buying and cutting vegetables yourself | Ordering a ready meal |
| Examples | Your survey, your app's sales records, an experiment | Government statistics, company reports, open datasets online |
| Strength | Fits your question exactly | Fast and cheap |
| Watch out | Slow and costly to collect | Check how it was collected and what each column means |
File formats
| Format | What it is | Text or binary | Python |
|---|---|---|---|
| CSV | Comma-separated values: one row per line, commas between values | Text | csv module, pd.read_csv |
| TSV | Tab-separated values: same idea, a tab between values | Text | csv with delimiter="\t", pd.read_csv(sep="\t") |
| JSON | JavaScript Object Notation: labels {} and lists [], nesting allowed |
Text | json module, pd.read_json |
| Excel (.xlsx) | A workbook with sheets, formatting and formulas | Binary (a zip file) | pd.read_excel (needs openpyxl) |
| Parquet | Compressed, stored column by column; fast and small for big data | Binary | pd.read_parquet (needs pyarrow) |
A delimiter is the character that separates values (a comma in CSV, a tab in TSV).
CSV and TSV
import csv
rows = [["name", "marks", "size"], ["Riya", 82, "M"], ["Aman", 75, "L"]]
with open("class.csv", "w", newline="") as f:
csv.writer(f).writerows(rows)
with open("class.tsv", "w", newline="") as f:
csv.writer(f, delimiter="\t").writerows(rows)
print(open("class.csv").read())
print(open("class.tsv").read())
name,marks,size
Riya,82,M
Aman,75,L
name marks size
Riya 82 M
Aman 75 L
JSON (semi-structured)
import json
riya = {"name": "Riya", "marks": {"maths": 82, "science": 88},
"phones": ["98200 11111", "98200 22222"]}
with open("riya.json", "w") as f:
json.dump(riya, f, indent=2)
print(open("riya.json").read())
{
"name": "Riya",
"marks": {
"maths": 82,
"science": 88
},
"phones": [
"98200 11111",
"98200 22222"
]
}
Curly brackets hold labelled values; square brackets hold lists. Riya's marks are nested inside her record, something a flat table can't hold neatly.
Parquet (binary)
import pandas as pd
df = pd.DataFrame({"name": ["Riya", "Aman"], "marks": [82, 75]})
df.to_parquet("class.parquet")
print(open("class.parquet").read())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 262: invalid start byte
Python tried to read the file as text, but Parquet is binary (raw, compressed bytes). Open it in binary mode ("rb"), or better, let pandas read it:
import pandas as pd
df = pd.DataFrame({"name": ["Riya", "Aman"], "marks": [82, 75]})
df.to_parquet("class.parquet")
print(open("class.parquet", "rb").read()[:40])
print(pd.read_parquet("class.parquet"))
b'PAR1\x15\x04\x15 \x15$L\x15\x04\x15\x00\x12\x00\x00\x10<\x04\x00\x00\x00Riya\x04\x00\x00\x00Aman\x15\x00\x15\x12'
name marks
0 Riya 82
1 Aman 75
Parquet files start with the signature PAR1. (pandas is the main data library of this course; episode 3 shows how to install it and episodes 11+ teach it.)
Excel is binary too
import pandas as pd
df = pd.DataFrame({"name": ["Riya", "Aman"], "marks": [82, 75]})
df.to_excel("class.xlsx", index=False)
print(open("class.xlsx", "rb").read()[:4])
print(pd.read_excel("class.xlsx"))
b'PK\x03\x04'
name marks
0 Riya 82
1 Aman 75
An .xlsx file starts with PK, the signature of a zip archive: inside it are several XML files.
Common mistakes (with the real error messages)
| Mistake | Real message | Why | Fix |
|---|---|---|---|
sum("Riya got 82...") |
TypeError: unsupported operand type(s) for +: 'int' and 'str' |
Text has no structure | Put the data in a table: one row per record |
| Averaging pin codes | No error: 356685.0 |
Digits used as labels are categorical | Count them instead |
Averaging sizes ["M", "L", "S"] |
TypeError: unsupported operand type(s) for +: 'int' and 'str' |
Categories can't be added | Count them, or rank ordinal ones |
sorted(["M", "L", "S"]) |
No error: ['L', 'M', 'S'] |
Alphabetical, not size order | sorted(sizes, key=order.get) |
| Saying 40 °C is "twice" 20 °C | No error: a wrong claim | Interval data has no true zero | Compare differences, not ratios |
open("class.parquet").read() |
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 ... |
Parquet/Excel are binary | pd.read_parquet, pd.read_excel |
Mini project: tidy up Ramesh's monthly sales
Turn the untidy table (months as columns) into tidy rows, then find the best stall-month.
wide = {"Andheri": {"Jan": 9100, "Feb": 8800, "Mar": 9400},
"Bandra": {"Jan": 8200, "Feb": 7900, "Mar": 8600}}
tidy = []
for stall, months in wide.items():
for month, cups in months.items():
tidy.append({"stall": stall, "month": month, "cups": cups})
for row in tidy:
print(row)
best = max(tidy, key=lambda r: r["cups"])
print("Best:", best)
{'stall': 'Andheri', 'month': 'Jan', 'cups': 9100}
{'stall': 'Andheri', 'month': 'Feb', 'cups': 8800}
{'stall': 'Andheri', 'month': 'Mar', 'cups': 9400}
{'stall': 'Bandra', 'month': 'Jan', 'cups': 8200}
{'stall': 'Bandra', 'month': 'Feb', 'cups': 7900}
{'stall': 'Bandra', 'month': 'Mar', 'cups': 8600}
Best: {'stall': 'Andheri', 'month': 'Mar', 'cups': 9400}
Once the data is tidy, "best month" is a single max over one column.
Practice
1. Classify each column (quantitative discrete / quantitative continuous / nominal / ordinal) and give its level of measurement: (a) number of UPI payments in a day, (b) amount paid in ₹, (c) bank name, (d) customer satisfaction: poor / fair / good / excellent, (e) year of birth, (f) mobile number.
Solution
(a) quantitative discrete, ratio. (b) quantitative continuous, ratio. (c) nominal. (d) ordinal. (e) quantitative, interval (year 0 isn't "no time"; differences make sense, ratios don't). (f) nominal: digits used as a label.
2. Is this table tidy? student, maths, science, english with one row per student. If not, what would the tidy version's columns be?
Solution
Not tidy for analysing marks: the variable subject hides in the headers. Tidy columns: student, subject, marks, one row per student per subject.
3. Save three students as JSON, load the file back and print the third student and the average.
Solution
import json
students = [{"name": "Riya", "marks": 82}, {"name": "Aman", "marks": 75},
{"name": "Sara", "marks": 91}]
with open("class.json", "w") as f:
json.dump(students, f)
loaded = json.load(open("class.json"))
print(loaded[2]["name"], loaded[2]["marks"])
print(sum(s["marks"] for s in loaded) / len(loaded))
Sara 91
82.66666666666667
4. Primary or secondary? (a) A Google Form you send to your college friends. (b) A dataset of IPL matches downloaded from the internet. (c) Your shop's billing app records.
Solution
(a) primary, (b) secondary, (c) primary (collected by your own business for its own use).
Quick quiz
A survey column customer_rating holds 1, 2, 3, 4 or 5 stars. What type of data is it?
A. Continuous B. Nominal C. Ordinal D. Ratio
Answer
C. Ordinal. 5 stars is better than 4, so there's an order, but the gap between 1★ and 2★ isn't a fixed amount. Be careful averaging ratings; many analysts report the median or the share of 5-star ratings.
Interview questions
- What's the difference between structured, semi-structured and unstructured data? Fixed table vs labelled/nested records (JSON) vs no fields (images, audio, free text).
- Nominal vs ordinal? Both are categories; ordinal ones have a natural order (S/M/L), nominal ones don't (city).
- Explain the four levels of measurement. Nominal, ordinal, interval (equal gaps, no true zero, e.g. °C), ratio (true zero, e.g. money).
- Is a pin code numerical data? No: it's categorical (nominal); its digits are a label, so averages are meaningless.
- What is tidy data and why does it matter? One variable per column, one observation per row, one value per cell; tools like pandas and plotting libraries expect it.
- CSV vs Parquet? CSV is plain text, row by row, readable anywhere; Parquet is binary, compressed and column-based, so it is smaller and much faster for large datasets.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Data | Recorded facts | "Riya, 82, M" |
| Dataset | A collection of data about one topic, usually a table | A class register |
| Observation / record / row | Everything recorded about one thing | Riya's line |
| Variable / feature / column | One fact recorded for every row | marks |
| Dictionary | Python values with labels | {"name": "Riya", "marks": 82} |
| Structured data | Fixed rows and columns | Bank statement |
| Semi-structured data | Labels and nesting, flexible fields | A JSON contact |
| Unstructured data | No fields at all | Photos, voice notes, reviews |
| Quantitative (numerical) data | An amount you can do maths on | Price |
| Qualitative (categorical) data | A label or group | City |
| Discrete | Counted, whole steps | Cups sold |
| Continuous | Measured, any value in between | Weight 61.37 kg |
| Nominal | Categories with no order | Blood group |
| Ordinal | Categories with a natural order | S < M < L |
| Levels of measurement | Nominal, ordinal, interval, ratio: how much maths a variable allows | — |
| Interval | Equal gaps, no true zero | °C |
| Ratio | Equal gaps and a true zero | ₹, kg |
| True zero | Zero means "none at all" | ₹0 |
| Tidy data | One variable per column, one observation per row, one value per cell | stall, month, cups |
| Primary data | Collected by you for your question | Your own survey |
| Secondary data | Collected by someone else | Government statistics |
| File format | How data is stored in a file | CSV, JSON |
| CSV | Comma-separated values (text) | Riya,82,M |
| TSV | Tab-separated values (text) | Riya 82 M |
| Delimiter | The character that separates values | , or a tab |
| JSON | JavaScript Object Notation: labels and nesting (text) | {"name": "Riya"} |
| Excel (.xlsx) | A spreadsheet workbook (binary zip) | class.xlsx |
| Parquet | A compressed, column-based binary format | class.parquet |
| Binary file | Raw bytes, not readable text | Parquet, Excel |
| Nested data | Data inside data | Marks per subject inside a student |
| UnicodeDecodeError | Python's error when bytes can't be read as text | Opening Parquet as text |
Summary
- A dataset is rows (observations / records) × columns (variables / features). Structure is what lets a computer calculate.
- Quantitative data is discrete or continuous; qualitative data is nominal or ordinal. Digits can be labels.
- Levels of measurement: nominal → ordinal → interval → ratio; "twice as much" needs a true zero.
- Tidy data: one variable per column, one observation per row, one value per cell.
- Data is structured, semi-structured or unstructured, and primary or secondary.
- CSV, TSV, JSON are text; Excel and Parquet are binary. Use the right reader.
- Next episode: setting up Jupyter, Colab, pip and virtual environments.