SyntaxPilot
Courses/Data Analytics with Python/Lesson 2
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Data Analytics with Python · Lesson 2 of 90

Data, datasets and types of data

datadatasetobservationrecordvariablefeaturestructured / semi-structured / unstructured dataquantitative vs qualitativenumerical (discrete, continuous)categorical (nominal, ordinal)levels of measurement (nominal, ordinal, interval, ratio)tidy dataprimary vs secondary datafile formats (CSV, TSV, JSON, Excel, Parquet)

These are the notes for Data Analytics with Python, episode 2. You will learn what a dataset is (observations and variables), the types of data (quantitative vs qualitative, discrete vs continuous, nominal vs ordinal), the four levels of measurement, tidy data, structured vs semi-structured vs unstructured data, primary vs secondary data, and the five file formats analysts meet every week. Every term is defined the first time it appears and collected in the Glossary. Every program runs as-is in the SyntaxPilot compiler, and every output is the real output from Python 3.13 (pandas and pyarrow installed).

The problem: marks stuck in a sentence

Mrs Iyer, a class teacher with 40 students and 6 subjects, writes marks in her diary as sentences: "Riya got 82 in maths, Aman got 75, Sara got 91". When the principal asks for the class average, she has to read every sentence again. Python can't do better with a sentence:

notes = "Riya got 82 in maths, Aman got 75, Sara got 91"
print(notes)
print(sum(notes))

Output:

Riya got 82 in maths, Aman got 75, Sara got 91
Traceback (most recent call last):
  ...
TypeError: unsupported operand type(s) for +: 'int' and 'str'

Why: a sentence has no structure. To Python it's just characters; nothing says which part is a name and which part is a mark.

Fix: a table, like a school register: one row per student, one column per fact.

Datasets: observations and variables

School register Data word Meaning
The whole register Dataset A collection of data about one topic, usually a table
One line (everything about Riya) Observation = record = row Everything recorded about one thing
One heading (Marks) Variable = feature = column One fact recorded for every row. It "varies" from row to row. Feature is the machine-learning word

In Python, each student can be a dictionary: values with labels ("marks": 82).

# each row = one student, each column = one variable
students = [
    {"name": "Riya", "marks": 82, "size": "M", "pin": 400053},
    {"name": "Aman", "marks": 75, "size": "L", "pin": 110001},
    {"name": "Sara", "marks": 91, "size": "S", "pin": 560001},
]
marks = [s["marks"] for s in students]
print("Average marks:", sum(marks) / len(marks))
pins = [s["pin"] for s in students]
print("Average pin code:", sum(pins) / len(pins))

Output:

Average marks: 82.66666666666667
Average pin code: 356685.0

The average of marks is meaningful. The average of pin codes ran without an error and means nothing: 356685 isn't a place. And the average of sizes crashes:

sizes = ["M", "L", "S"]
print(sum(sizes) / len(sizes))
TypeError: unsupported operand type(s) for +: 'int' and 'str'

Rule 1 of analytics: before you calculate anything, ask what type of data the column holds. No error does not mean a correct answer.

Types of data

Data
├── Quantitative (numerical): an amount you can do maths on
│   ├── Discrete: counted, whole steps        → cups sold, number of children
│   └── Continuous: measured, any value       → weight 61.37 kg, time 9.58 s
└── Qualitative (categorical): a label / group
    ├── Nominal: no order                      → blood group, city, pin code
    └── Ordinal: a natural order               → S < M < L, 1–5 star rating
Type Plain meaning Examples Sensible maths
Quantitative (numerical) An amount marks, price, height sum, average, …
Discrete Counted; jumps in whole steps cups sold, students present counts, averages
Continuous Measured; any value in between weight, time, temperature averages, ranges
Qualitative (categorical) A label that puts a row in a group city, size, blood group counting per group
Nominal Categories with no order blood group A/B/O, pin code counts, the most common one
Ordinal Categories with a natural order S/M/L, star ratings, class rank order, median

Digits don't make a number: pin codes, phone numbers and roll numbers are categorical (nominal). Count them, don't average them:

pins = ["400053", "110001", "400053", "560001"]
counts = {}
for p in pins:
    counts[p] = counts.get(p, 0) + 1
print(counts)
{'400053': 2, '110001': 1, '560001': 1}

The ordinal trap: sorting sizes

sizes = ["M", "L", "S", "M"]
print(sorted(sizes))
order = {"S": 1, "M": 2, "L": 3}
print(sorted(sizes, key=order.get))
['L', 'M', 'M', 'S']
['S', 'M', 'M', 'L']

Python sorts letters alphabetically; it doesn't know small comes before large. Tell it the order with a dictionary that maps each size to a rank. (pandas has an ordered categorical type for this; we'll use it later in the course.)

The four levels of measurement

Each level allows more maths than the one before.

Level Plain meaning Example Allowed
Nominal Names only, no order Blood group, city = and ≠, counting
Ordinal An order, but gaps aren't equal Hotel stars, S/M/L also < and >, median
Interval Equal gaps, but no true zero Temperature in °C, calendar year also + and − (differences)
Ratio Equal gaps and a true zero (0 = none) Money, weight, cups, age everything, including "twice as much"

Proof that "40 °C is twice as hot as 20 °C" is false:

def to_f(c):
    return c * 9 / 5 + 32
print("Celsius:", 40 / 20)
print("Fahrenheit:", to_f(40) / to_f(20))
print("Rupees:", 40 / 20)
Celsius: 2.0
Fahrenheit: 1.5294117647058822
Rupees: 2.0

The ratio changes when only the unit changes (2 vs 1.53), so it was never a real fact. 0 °C doesn't mean "no heat". ₹40 really is twice ₹20, because ₹0 really means no money.

Tidy data

Same numbers, two layouts:

Untidy (months as columns):

stall,Jan,Feb,Mar
Andheri,9100,8800,9400
Bandra,8200,7900,8600

Tidy (one row per observation):

stall,month,cups
Andheri,Jan,9100
Andheri,Feb,8800
Andheri,Mar,9400
Bandra,Jan,8200
Bandra,Feb,7900
Bandra,Mar,8600

Tidy data rules: every column is one variable, every row is one observation, every cell holds one value. In the untidy table the variable month hides in the headers: adding April means adding a column, and "best month" means checking every column. Python and pandas tools expect tidy tables (later we'll switch shapes with one line of pandas: melt and pivot).

Structured, semi-structured and unstructured data

Kind Plain meaning Everyday example Typical files
Structured Fixed rows and columns, same fields for every row Bank statement: date, amount, balance CSV, Excel, database tables
Semi-structured Labels and nesting, but fields can differ per record A phone contact: two numbers for one person, email for another JSON, XML
Unstructured No fields at all Photos, voice notes, written reviews images, audio, free text

Unstructured data has to be processed (for example, reading the stars and keywords out of reviews) before an analyst can count it.

Primary vs secondary data

Primary data Secondary data
Who collected it You, for your own question Someone else, for another purpose
Kitchen analogy Buying and cutting vegetables yourself Ordering a ready meal
Examples Your survey, your app's sales records, an experiment Government statistics, company reports, open datasets online
Strength Fits your question exactly Fast and cheap
Watch out Slow and costly to collect Check how it was collected and what each column means

File formats

Format What it is Text or binary Python
CSV Comma-separated values: one row per line, commas between values Text csv module, pd.read_csv
TSV Tab-separated values: same idea, a tab between values Text csv with delimiter="\t", pd.read_csv(sep="\t")
JSON JavaScript Object Notation: labels {} and lists [], nesting allowed Text json module, pd.read_json
Excel (.xlsx) A workbook with sheets, formatting and formulas Binary (a zip file) pd.read_excel (needs openpyxl)
Parquet Compressed, stored column by column; fast and small for big data Binary pd.read_parquet (needs pyarrow)

A delimiter is the character that separates values (a comma in CSV, a tab in TSV).

CSV and TSV

import csv
rows = [["name", "marks", "size"], ["Riya", 82, "M"], ["Aman", 75, "L"]]
with open("class.csv", "w", newline="") as f:
    csv.writer(f).writerows(rows)
with open("class.tsv", "w", newline="") as f:
    csv.writer(f, delimiter="\t").writerows(rows)
print(open("class.csv").read())
print(open("class.tsv").read())
name,marks,size
Riya,82,M
Aman,75,L

name    marks   size
Riya    82  M
Aman    75  L

JSON (semi-structured)

import json
riya = {"name": "Riya", "marks": {"maths": 82, "science": 88},
        "phones": ["98200 11111", "98200 22222"]}
with open("riya.json", "w") as f:
    json.dump(riya, f, indent=2)
print(open("riya.json").read())
{
  "name": "Riya",
  "marks": {
    "maths": 82,
    "science": 88
  },
  "phones": [
    "98200 11111",
    "98200 22222"
  ]
}

Curly brackets hold labelled values; square brackets hold lists. Riya's marks are nested inside her record, something a flat table can't hold neatly.

Parquet (binary)

import pandas as pd
df = pd.DataFrame({"name": ["Riya", "Aman"], "marks": [82, 75]})
df.to_parquet("class.parquet")
print(open("class.parquet").read())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 262: invalid start byte

Python tried to read the file as text, but Parquet is binary (raw, compressed bytes). Open it in binary mode ("rb"), or better, let pandas read it:

import pandas as pd
df = pd.DataFrame({"name": ["Riya", "Aman"], "marks": [82, 75]})
df.to_parquet("class.parquet")
print(open("class.parquet", "rb").read()[:40])
print(pd.read_parquet("class.parquet"))
b'PAR1\x15\x04\x15 \x15$L\x15\x04\x15\x00\x12\x00\x00\x10<\x04\x00\x00\x00Riya\x04\x00\x00\x00Aman\x15\x00\x15\x12'
   name  marks
0  Riya     82
1  Aman     75

Parquet files start with the signature PAR1. (pandas is the main data library of this course; episode 3 shows how to install it and episodes 11+ teach it.)

Excel is binary too

import pandas as pd
df = pd.DataFrame({"name": ["Riya", "Aman"], "marks": [82, 75]})
df.to_excel("class.xlsx", index=False)
print(open("class.xlsx", "rb").read()[:4])
print(pd.read_excel("class.xlsx"))
b'PK\x03\x04'
   name  marks
0  Riya     82
1  Aman     75

An .xlsx file starts with PK, the signature of a zip archive: inside it are several XML files.

Common mistakes (with the real error messages)

Mistake Real message Why Fix
sum("Riya got 82...") TypeError: unsupported operand type(s) for +: 'int' and 'str' Text has no structure Put the data in a table: one row per record
Averaging pin codes No error: 356685.0 Digits used as labels are categorical Count them instead
Averaging sizes ["M", "L", "S"] TypeError: unsupported operand type(s) for +: 'int' and 'str' Categories can't be added Count them, or rank ordinal ones
sorted(["M", "L", "S"]) No error: ['L', 'M', 'S'] Alphabetical, not size order sorted(sizes, key=order.get)
Saying 40 °C is "twice" 20 °C No error: a wrong claim Interval data has no true zero Compare differences, not ratios
open("class.parquet").read() UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 ... Parquet/Excel are binary pd.read_parquet, pd.read_excel

Mini project: tidy up Ramesh's monthly sales

Turn the untidy table (months as columns) into tidy rows, then find the best stall-month.

wide = {"Andheri": {"Jan": 9100, "Feb": 8800, "Mar": 9400},
        "Bandra":  {"Jan": 8200, "Feb": 7900, "Mar": 8600}}
tidy = []
for stall, months in wide.items():
    for month, cups in months.items():
        tidy.append({"stall": stall, "month": month, "cups": cups})
for row in tidy:
    print(row)
best = max(tidy, key=lambda r: r["cups"])
print("Best:", best)
{'stall': 'Andheri', 'month': 'Jan', 'cups': 9100}
{'stall': 'Andheri', 'month': 'Feb', 'cups': 8800}
{'stall': 'Andheri', 'month': 'Mar', 'cups': 9400}
{'stall': 'Bandra', 'month': 'Jan', 'cups': 8200}
{'stall': 'Bandra', 'month': 'Feb', 'cups': 7900}
{'stall': 'Bandra', 'month': 'Mar', 'cups': 8600}
Best: {'stall': 'Andheri', 'month': 'Mar', 'cups': 9400}

Once the data is tidy, "best month" is a single max over one column.

Practice

1. Classify each column (quantitative discrete / quantitative continuous / nominal / ordinal) and give its level of measurement: (a) number of UPI payments in a day, (b) amount paid in ₹, (c) bank name, (d) customer satisfaction: poor / fair / good / excellent, (e) year of birth, (f) mobile number.

Solution

(a) quantitative discrete, ratio. (b) quantitative continuous, ratio. (c) nominal. (d) ordinal. (e) quantitative, interval (year 0 isn't "no time"; differences make sense, ratios don't). (f) nominal: digits used as a label.

2. Is this table tidy? student, maths, science, english with one row per student. If not, what would the tidy version's columns be?

Solution

Not tidy for analysing marks: the variable subject hides in the headers. Tidy columns: student, subject, marks, one row per student per subject.

3. Save three students as JSON, load the file back and print the third student and the average.

Solution
import json
students = [{"name": "Riya", "marks": 82}, {"name": "Aman", "marks": 75},
            {"name": "Sara", "marks": 91}]
with open("class.json", "w") as f:
    json.dump(students, f)
loaded = json.load(open("class.json"))
print(loaded[2]["name"], loaded[2]["marks"])
print(sum(s["marks"] for s in loaded) / len(loaded))
Sara 91
82.66666666666667

4. Primary or secondary? (a) A Google Form you send to your college friends. (b) A dataset of IPL matches downloaded from the internet. (c) Your shop's billing app records.

Solution

(a) primary, (b) secondary, (c) primary (collected by your own business for its own use).

Quick quiz

A survey column customer_rating holds 1, 2, 3, 4 or 5 stars. What type of data is it?

A. Continuous B. Nominal C. Ordinal D. Ratio

Answer

C. Ordinal. 5 stars is better than 4, so there's an order, but the gap between 1★ and 2★ isn't a fixed amount. Be careful averaging ratings; many analysts report the median or the share of 5-star ratings.

Interview questions

  1. What's the difference between structured, semi-structured and unstructured data? Fixed table vs labelled/nested records (JSON) vs no fields (images, audio, free text).
  2. Nominal vs ordinal? Both are categories; ordinal ones have a natural order (S/M/L), nominal ones don't (city).
  3. Explain the four levels of measurement. Nominal, ordinal, interval (equal gaps, no true zero, e.g. °C), ratio (true zero, e.g. money).
  4. Is a pin code numerical data? No: it's categorical (nominal); its digits are a label, so averages are meaningless.
  5. What is tidy data and why does it matter? One variable per column, one observation per row, one value per cell; tools like pandas and plotting libraries expect it.
  6. CSV vs Parquet? CSV is plain text, row by row, readable anywhere; Parquet is binary, compressed and column-based, so it is smaller and much faster for large datasets.

Glossary

Term Meaning Example
Data Recorded facts "Riya, 82, M"
Dataset A collection of data about one topic, usually a table A class register
Observation / record / row Everything recorded about one thing Riya's line
Variable / feature / column One fact recorded for every row marks
Dictionary Python values with labels {"name": "Riya", "marks": 82}
Structured data Fixed rows and columns Bank statement
Semi-structured data Labels and nesting, flexible fields A JSON contact
Unstructured data No fields at all Photos, voice notes, reviews
Quantitative (numerical) data An amount you can do maths on Price
Qualitative (categorical) data A label or group City
Discrete Counted, whole steps Cups sold
Continuous Measured, any value in between Weight 61.37 kg
Nominal Categories with no order Blood group
Ordinal Categories with a natural order S < M < L
Levels of measurement Nominal, ordinal, interval, ratio: how much maths a variable allows —
Interval Equal gaps, no true zero °C
Ratio Equal gaps and a true zero ₹, kg
True zero Zero means "none at all" ₹0
Tidy data One variable per column, one observation per row, one value per cell stall, month, cups
Primary data Collected by you for your question Your own survey
Secondary data Collected by someone else Government statistics
File format How data is stored in a file CSV, JSON
CSV Comma-separated values (text) Riya,82,M
TSV Tab-separated values (text) Riya 82 M
Delimiter The character that separates values , or a tab
JSON JavaScript Object Notation: labels and nesting (text) {"name": "Riya"}
Excel (.xlsx) A spreadsheet workbook (binary zip) class.xlsx
Parquet A compressed, column-based binary format class.parquet
Binary file Raw bytes, not readable text Parquet, Excel
Nested data Data inside data Marks per subject inside a student
UnicodeDecodeError Python's error when bytes can't be read as text Opening Parquet as text

Summary

  • A dataset is rows (observations / records) × columns (variables / features). Structure is what lets a computer calculate.
  • Quantitative data is discrete or continuous; qualitative data is nominal or ordinal. Digits can be labels.
  • Levels of measurement: nominal → ordinal → interval → ratio; "twice as much" needs a true zero.
  • Tidy data: one variable per column, one observation per row, one value per cell.
  • Data is structured, semi-structured or unstructured, and primary or secondary.
  • CSV, TSV, JSON are text; Excel and Parquet are binary. Use the right reader.
  • Next episode: setting up Jupyter, Colab, pip and virtual environments.