These are the notes for Python Coding Questions, episode 28: file and data questions. Real programs read real files, and real files are messy. Three interview questions: count the words in a file, find the top-k frequent words, and total a column of a CSV file that has bad rows.
Every program was run with Python 3.13 (python3 main.py), the SyntaxPilot compiler's Python command. Every output below is real. Timings change from machine to machine and run to run; what matters is how they compare.
Words for today:
- File path: the name (and folder) of a file, like
review.txt. A bare name means "in the program's current folder". - Mode: how a file is opened:
"r"read (the default),"w"write (creates or empties it),"a"append. - Context manager: an object used with
withthat sets something up and always cleans up;open()closes the file at the end of thewithblock, even after an error. - Whitespace: spaces, tabs and new lines.
- Regular expression (regex): a pattern for text;
re.findall(pattern, text)returns every match. - Top-k: the k biggest items by some value.
- Heap: a structure that always gives the smallest item fast;
heapq.nlargest(k, ...)uses one to keep a shortlist of k. - CSV: comma separated values, a spreadsheet saved as plain text. Values that contain commas are wrapped in quotes.
Q1: Count the words in a file
Real-life example: a café saves each customer review in a text file. The owner wants the number of words, and how often each word appears.
Attempt 1: open the file and split on spaces
f = open("review.txt")
text = f.read()
print(len(text.split(" ")))
Output:
Traceback (most recent call last):
File "main.py", line 1, in <module>
f = open("review.txt")
FileNotFoundError: [Errno 2] No such file or directory: 'review.txt'
FileNotFoundError: [Errno 2] No such file or directory: 'review.txt'. The file is looked up in the program's current folder, and it isn't there. Check the name and the folder, or (as here) create it first.
Writing and reading safely with with
with open("review.txt", "w") as f:
f.write(review)
with closes the file at the end of the block, even if an error happens inside. Without it you must call f.close() yourself; forget it and written data may still sit in a memory buffer, not on the disk.
review = """Great chai. Great samosa!
The chai was hot,
the samosa was crisp.
"""
with open("review.txt", "w") as f: # closed for us at the end
f.write(review)
print("closed?", f.closed)
with open("review.txt") as f:
text = f.read()
print(len(text.split(" "))) # split on single spaces
print(text.split(" ")[:5])
print(len(text.split())) # split on any whitespace
Output:
closed? True
11
['Great', 'chai.', '', 'Great', 'samosa!\nThe']
12
f.closed is True after the block. But split(" ") gives 11, while split() gives 12: split(" ") cuts only at single spaces, so the double space makes an empty string '', and the new line glues 'samosa!\nThe' into one piece. split() with no argument cuts at any run of whitespace and drops empty pieces.
Counting each word
Plain split() still treats "Great" and "great", "chai." and "chai" as different words. Clean each line first: line.lower(), then re.findall(r"[a-z']+", ...) keeps only runs of letters (and apostrophes, for words like don't). Count with a dictionary, and read the file line by line (for line in f) so even a huge file never has to fit in memory.
import re
with open("review.txt", "w") as f:
f.write("Great chai. Great samosa!\nThe chai was hot,\n"
"the samosa was crisp.\n")
def count_words(path):
counts = {}
with open(path) as f:
for line in f: # one line at a time
for word in re.findall(r"[a-z']+", line.lower()):
counts[word] = counts.get(word, 0) + 1
return counts
counts = count_words("review.txt")
print(sum(counts.values()), "words,", len(counts), "different")
print(counts)
Output:
12 words, 7 different
{'great': 2, 'chai': 2, 'samosa': 2, 'the': 2, 'was': 2, 'hot': 1, 'crisp': 1}
O(n) time for n characters; memory is one line plus one entry per different word.
| Way to split | Result on "Great chai. Great samosa!\nThe ..." |
|---|---|
text.split(" ") |
'chai.', '', 'samosa!\nThe': wrong count |
text.split() |
'chai.', 'samosa!', 'The': right count, punctuation kept |
re.findall(r"[a-z']+", text.lower()) |
'chai', 'samosa', 'the': clean words |
Q2: Top-k frequent words
Real-life example: a college canteen writes down every order. The manager wants the 3 most ordered snacks for the menu board.
Attempt 1: sorted(counts)[:3]
words = ("chai samosa chai vada chai samosa dosa "
"vada chai idli samosa dosa chai").split()
counts = {}
for w in words:
counts[w] = counts.get(w, 0) + 1
print(counts)
print(sorted(counts)[:3]) # top 3?
Output:
{'chai': 5, 'samosa': 3, 'vada': 2, 'dosa': 2, 'idli': 1}
['chai', 'dosa', 'idli']
The counts are right, but the "top 3" is chai, dosa, idli, and idli was ordered once. Sorting a dictionary sorts its keys (the words, alphabetically); it never looks at the counts.
The fix: sort by the count, or keep a shortlist
counts.items()gives(word, count)pairs.sorted(pairs, key=lambda kv: kv[1], reverse=True)[:k]sorts by the count: O(n log n).heapq.nlargest(k, pairs, key=...)keeps only a heap of the k best so far: O(n log k).collections.Counter(words)is a dictionary that counts for you;.most_common(k)gives the top k.- Ties keep the order the words were first seen in (all three are stable).
import heapq, random, time
from collections import Counter
words = ("chai samosa chai vada chai samosa dosa "
"vada chai idli samosa dosa chai").split()
counts = Counter(words) # a dict that counts
pairs = counts.items()
by_count = sorted(pairs, key=lambda kv: kv[1], reverse=True)
print(by_count[:3])
print(heapq.nlargest(3, pairs, key=lambda kv: kv[1]))
print(counts.most_common(3))
print(heapq.nlargest(10, Counter(["a"]).items()))
big = Counter({f"w{i}": random.randint(1, 10**6)
for i in range(10**6)}) # a million words
start = time.perf_counter()
sorted(big.items(), key=lambda kv: kv[1], reverse=True)[:5]
print("sorted ", f"{time.perf_counter() - start:.2f} s")
start = time.perf_counter()
heapq.nlargest(5, big.items(), key=lambda kv: kv[1])
print("nlargest", f"{time.perf_counter() - start:.2f} s")
Output:
[('chai', 5), ('samosa', 3), ('vada', 2)]
[('chai', 5), ('samosa', 3), ('vada', 2)]
[('chai', 5), ('samosa', 3), ('vada', 2)]
[('a', 1)]
sorted 1.84 s
nlargest 0.12 s
All three give chai 5, samosa 3, vada 2 (vada and dosa both have 2; vada was seen first). Asking for more than there are just returns what exists. On a million words, nlargest is more than ten times faster than sorting everything.
| Way | Time | When |
|---|---|---|
sorted(..., key=...)[:k] |
O(n log n) | you also want the full ranking |
heapq.nlargest(k, ..., key=...) |
O(n log k) | k is much smaller than n |
Counter(...).most_common(k) |
uses a heap for k | you are counting anyway |
Q3: Parse a CSV file and total a column
Real-life example: a canteen's sales sheet with item, qty, price. The day's total is qty × price summed over every row. The sheet was typed by hand: "Samosa, large" has a comma inside quotes, vada pav's quantity is N/A, and the dosa row has no price.
Attempt 1: split each line on commas
data = """item,qty,price
Chai,2,15
"Samosa, large",3,20
Vada pav,N/A,25
Dosa,1
Idli,4,10
"""
with open("sales.csv", "w") as f:
f.write(data)
total = 0
with open("sales.csv") as f:
for line in f:
item, qty, price = line.strip().split(",")
total += int(qty) * float(price)
print("total:", total)
Output:
Traceback (most recent call last):
File "main.py", line 15, in <module>
total += int(qty) * float(price)
~~~^^^^^
ValueError: invalid literal for int() with base 10: 'qty'
ValueError: invalid literal for int() with base 10: 'qty': the first line is the header (the column names). Skip it with next(f), which reads one line and throws it away:
data = """item,qty,price
Chai,2,15
"Samosa, large",3,20
Vada pav,N/A,25
Dosa,1
Idli,4,10
"""
with open("sales.csv", "w") as f:
f.write(data)
total = 0
with open("sales.csv") as f:
next(f) # skip the header line
for line in f:
item, qty, price = line.strip().split(",")
total += int(qty) * float(price)
print("total:", total)
Output:
Traceback (most recent call last):
File "main.py", line 15, in <module>
item, qty, price = line.strip().split(",")
^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 3)
ValueError: too many values to unpack (expected 3): "Samosa, large" was cut into 4 pieces. CSV has rules (quotes protect commas, and can even hold new lines) and split(",") ignores them.
The csv module
Open CSV files with newline="" (the csv documentation asks for it, so the module handles line endings itself). csv.reader turns each line into a list of strings, respecting quotes:
import csv
data = """item,qty,price
Chai,2,15
"Samosa, large",3,20
Vada pav,N/A,25
Dosa,1
Idli,4,10
"""
with open("sales.csv", "w", newline="") as f:
f.write(data)
with open("sales.csv", newline="") as f:
for row in csv.reader(f):
print(row)
Output:
['item', 'qty', 'price']
['Chai', '2', '15']
['Samosa, large', '3', '20']
['Vada pav', 'N/A', '25']
['Dosa', '1']
['Idli', '4', '10']
The fix: csv.DictReader, named exceptions, and a report
csv.DictReaderuses the header as keys:row["qty"]instead ofrow[1], and it keeps working if the columns move. A short row getsNonefor the missing values.int("N/A")raisesValueError;float(None)raisesTypeError.- Catch exactly those errors, note the line number and item, and carry on. Line numbers start at 2 because line 1 is the header.
import csv
with open("sales.csv", "w", newline="") as f:
f.write('item,qty,price\nChai,2,15\n"Samosa, large",3,20\n'
'Vada pav,N/A,25\nDosa,1\nIdli,4,10\n')
def total_sales(path):
total, bad = 0, []
with open(path, newline="") as f:
for n, row in enumerate(csv.DictReader(f), start=2):
try:
total += int(row["qty"]) * float(row["price"])
except (ValueError, TypeError) as e:
bad.append((n, row["item"], type(e).__name__))
return total, bad
total, bad = total_sales("sales.csv")
print("total:", total)
for line in bad:
print("skipped line", *line)
Output:
total: 130.0
skipped line 4 Vada pav ValueError
skipped line 5 Dosa TypeError
Total 130.0 (chai 30 + samosa 60 + idli 40), and the two bad lines are reported, not silently lost. O(rows) time.
Common mistakes (and what you really see)
| Mistake | What you see | Why | Fix |
|---|---|---|---|
open("review.txt") on a missing file |
FileNotFoundError: [Errno 2] No such file or directory: 'review.txt' |
wrong folder or name | check the path; create it first |
f = open(...) and no close() |
data may never reach the disk | the file is never closed | with open(...) as f: |
text.split(" ") |
11 words instead of 12 | '' pieces and words glued by \n |
text.split() |
| counting raw words | Great and great counted apart |
case and punctuation | lower() + re.findall |
sorted(counts)[:3] |
['chai', 'dosa', 'idli'] |
sorts the keys | key=lambda kv: kv[1] or key=counts.get |
line.split(",") on a CSV |
ValueError: too many values to unpack (expected 3) |
quotes protect commas | csv.reader / csv.DictReader |
| forgetting the header | ValueError: invalid literal for int() with base 10: 'qty' |
line 1 is column names | next(f) or DictReader |
except: pass |
bad rows (and your bugs) vanish silently | catches everything | except (ValueError, TypeError) and report |
Sorting a dict sorts its keys:
counts = {"chai": 5, "samosa": 3, "vada": 2, "dosa": 2, "idli": 1}
print(sorted(counts))
print(sorted(counts, key=counts.get, reverse=True)[:3])
Output:
['chai', 'dosa', 'idli', 'samosa', 'vada']
['chai', 'samosa', 'vada']
A bare except: pass even hides a typo (prce), so the total is silently 0:
rows = [["Chai", "2", "15"], ["Vada pav", "N/A", "25"]]
total = 0
for item, qty, price in rows:
try:
total += int(qty) * float(prce) # a typo: prce
except:
pass # hides EVERYTHING
print("total:", total)
Output:
total: 0
Your turn: the best seller (solution)
Question: from the same sales.csv, return the item that earned the most money (qty × price), skipping bad rows.
Idea: DictReader, a dictionary of totals per item, then max(earned, key=earned.get): the key whose value is biggest.
import csv
with open("sales.csv", "w", newline="") as f:
f.write('item,qty,price\nChai,2,15\n"Samosa, large",3,20\n'
'Vada pav,N/A,25\nDosa,1\nIdli,4,10\n')
def best_item(path):
earned = {}
with open(path, newline="") as f:
for row in csv.DictReader(f):
try:
money = int(row["qty"]) * float(row["price"])
except (ValueError, TypeError):
continue # skip bad rows
earned[row["item"]] = earned.get(row["item"], 0) + money
if not earned:
return None
best = max(earned, key=earned.get) # the key with the biggest value
return best, earned[best]
print(best_item("sales.csv"))
Output:
('Samosa, large', 60.0)
Mini project: a review report
All three ideas together: read a CSV of reviews, skip rows with bad star ratings, average the stars, and find the top words without common "stop words".
import csv, heapq, re
from collections import Counter
with open("reviews.csv", "w", newline="") as f:
f.write('id,stars,text\n'
'1,5,"Great chai, great samosa!"\n'
'2,4,The chai was hot and the vada was crisp\n'
'3,five,"Samosa cold, chai great"\n'
'4,2,"Too sweet. The chai, not the samosa."\n')
STOP = {"the", "a", "and", "was", "not", "too"}
def review_report(path, k=3):
words, stars, bad = Counter(), [], []
with open(path, newline="") as f:
for n, row in enumerate(csv.DictReader(f), start=2):
try:
stars.append(int(row["stars"]))
except ValueError:
bad.append(n)
continue
words.update(w for w in re.findall(r"[a-z']+", row["text"].lower())
if w not in STOP)
avg = sum(stars) / len(stars) if stars else None
return avg, heapq.nlargest(k, words.items(), key=lambda kv: kv[1]), bad
avg, top, bad = review_report("reviews.csv")
print(f"average stars: {avg:.2f}")
print("top words:", top)
print("bad lines:", bad)
Output:
average stars: 3.67
top words: [('chai', 3), ('great', 2), ('samosa', 2)]
bad lines: [4]
Practice questions (with solutions)
1. Your own wc: count the lines, words and characters of a file.
Solution
def wc(path):
lines = words = chars = 0
with open(path) as f:
for line in f:
lines += 1
words += len(line.split())
chars += len(line)
return lines, words, chars
with open("poem.txt", "w") as f:
f.write("Chai in the morning,\nchai at night.\n\nChai makes everything right.\n")
print(wc("poem.txt"))
Output:
(4, 11, 67)
4 lines (the empty line counts), 11 words (split() ignores the double space), 67 characters including the new lines.
2. Merge two CSV files by item: prices from one file, stock from another.
Solution
Load the smaller file into a dictionary, then look each row of the other one up: O(n + m). A missing price comes back as None from dict.get.
import csv
with open("prices.csv", "w", newline="") as f:
f.write("item,price\nChai,15\nSamosa,20\nIdli,10\n")
with open("stock.csv", "w", newline="") as f:
f.write("item,stock\nChai,40\nIdli,25\nDosa,12\n")
def merge(prices_path, stock_path):
with open(prices_path, newline="") as f:
prices = {r["item"]: float(r["price"]) for r in csv.DictReader(f)}
merged = []
with open(stock_path, newline="") as f:
for r in csv.DictReader(f):
merged.append((r["item"], prices.get(r["item"]), int(r["stock"])))
return merged
for row in merge("prices.csv", "stock.csv"):
print(row)
Output:
('Chai', 15.0, 40)
('Idli', 10.0, 25)
('Dosa', None, 12)
3. Top-k words without stop words like "the", "was", "and".
Solution
import re
from collections import Counter
STOP = {"the", "a", "an", "and", "was", "is", "of"}
def top_words(text, k):
words = re.findall(r"[a-z']+", text.lower())
return Counter(w for w in words if w not in STOP).most_common(k)
text = "The chai was hot and the samosa was crisp. The chai is the best chai of the city."
print(top_words(text, 3))
Output:
[('chai', 3), ('hot', 1), ('samosa', 1)]
Quick quiz
What does this print?
text = "tea tea\ncoffee"
print(len(text.split(" ")), len(text.split()))
(A) 3 3 (B) 4 3 (C) 3 2 (D) 4 2
Answer
(A). Same count, different pieces: only split() gives the real words.
text = "tea tea\ncoffee"
print(len(text.split(" ")), len(text.split()))
print(text.split(" "), text.split())
Output:
3 3
['tea', '', 'tea\ncoffee'] ['tea', 'tea', 'coffee']
Interview questions
- Why use
with open(...)? It closes the file at the end of the block even if an exception is raised, so data is flushed and the file handle isn't leaked. - Difference between
"w"and"a"?"w"creates or empties the file;"a"appends to the end. - How do you read a 10 GB file? Line by line (
for line in f) or in fixed-size chunks (f.read(65536)), neverf.read()all at once. split()vssplit(" ")?split()splits on any run of whitespace and drops empty strings;split(" ")splits on every single space.- How do you sort a dict by value?
sorted(d.items(), key=lambda kv: kv[1], reverse=True)orsorted(d, key=d.get, reverse=True). heapq.nlargestvs sorting? O(n log k) vs O(n log n);nlargestwins when k is small.- What does
Counter.most_common(k)return? A list of(item, count)pairs, biggest counts first. - Why not split CSV lines on commas? Quoted values may contain commas (and new lines); the
csvmodule follows the format's rules. - Why
newline=""when opening CSV files? So the csv module sees the raw line endings and handles quoted new lines correctly. - How should a data job handle bad rows? Catch the specific errors, skip the row, and report it (line number and reason), so the result can be trusted and the data fixed.
Glossary
| Term | Meaning | Example |
|---|---|---|
| File path | Name and folder of a file | review.txt |
FileNotFoundError |
Raised when the file doesn't exist | [Errno 2] No such file or directory |
| Mode | How a file is opened | "r", "w", "a" |
with / context manager |
Sets up and always cleans up | the file is closed after the block |
f.closed |
True once the file is closed |
after with |
| Whitespace | Spaces, tabs, new lines | "\n" |
split() |
Split on any whitespace | "a b".split() → ['a', 'b'] |
| Regex | A text pattern | [a-z']+ |
re.findall |
Every match of a pattern | the words of a line |
lower() |
Lower-case copy of a string | "Great" → "great" |
dict.get(k, 0) |
The value, or 0 if missing | counting |
Counter |
A dict that counts | Counter(words) |
most_common(k) |
Top k (item, count) pairs |
[('chai', 5), ...] |
| Top-k | The k biggest by some value | 3 snacks |
Heap / heapq |
Fast access to the smallest item | heapq.nlargest |
key= |
What to sort or compare by | key=lambda kv: kv[1] |
lambda |
A small one-line function | lambda kv: kv[1] |
| Stable sort | Equal items keep their order | vada before dosa |
| CSV | Comma separated values | item,qty,price |
| Header | The first row: column names | item,qty,price |
next(f) |
Read (and skip) one line | skip the header |
csv.reader |
Rows as lists, quotes respected | ['Samosa, large', '3', '20'] |
csv.DictReader |
Rows as dicts keyed by the header | row["qty"] |
ValueError |
Right type, bad value | int("N/A") |
TypeError |
Wrong type | float(None) |
Bare except |
Catches every error | hides bugs |
| Stop words | Very common words left out of counts | the, was, and |
Summary
- Files:
with open(path, mode) as f:always closes the file;FileNotFoundErrormeans a wrong name or folder; read big files line by line. - Words:
split(), notsplit(" "); clean withlower()andre.findall(r"[a-z']+", ...); count with a dict orCounter. - Top-k:
sorted(dict)sorts keys; sort the pairs by count, or useheapq.nlargest(O(n log k)) orCounter.most_common. - CSV: use the
csvmodule (newline="",DictReader), catch the errors you expect (ValueError,TypeError), and report skipped rows. - Next episode: the machine coding round: a key-value store with TTL and an event emitter, tested with
unittest.