These are the notes for Python episode 24. You've written hundreds of for loops; in this episode you find out what a for loop actually does, and you invent generators: functions that hand out values one at a time, so you can process millions of items with almost no memory. Raju's chai business now has ten shops and ten million bills a year. Building all of them in a list just to add them up eats hundreds of megabytes. We'll cover iterables vs iterators, iter() and next(), StopIteration, writing an iterator class with __iter__ and __next__, yield and how a generator pauses and resumes, generator expressions, the used-up generator trap, endless generators, and a lazy pipeline that processes a file one line at a time. Every example below runs as-is in the SyntaxPilot compiler, with its real output from Python 3.13.
The problem: ten million bills
Raju wants the total of ten million bills. The obvious way, the one we've used so far, builds a full list and then sums it. Python's tracemalloc module can tell us the peak memory our program used:
import tracemalloc
def all_bills(n):
bills = []
for i in range(n):
bills.append(i * 2 + 10)
return bills
tracemalloc.start()
total = sum(all_bills(1_000_000))
print("Total:", total)
peak = tracemalloc.get_traced_memory()[1]
print(f"Peak memory: {peak:,} bytes")
Output:
Total: 1000009000000
Peak memory: 40,444,808 bytes
(tracemalloc, short for trace memory allocations, records every block of memory Python allocates; get_traced_memory() returns the current and the peak amount in bytes. Tracing slows Python down, so we use one million bills instead of ten million; ten million would need about ten times as much. 1_000_000 is just 1000000: underscores in numbers are allowed for readability.)
About 40 MB for one million bills (so about 400 MB for ten million), just to add up numbers.
Why it fails: the list keeps all ten million numbers alive at the same time. But sum only ever needs one number at a time: add it, forget it. It's like a clerk who prints every bill and stacks them all on the desk before he starts adding: the desk fills up long before he's done.
What we want: a function that hands out one bill, waits, hands out the next, and never keeps the pile. To build it, we first need to see how a for loop asks for items.
How a for loop really works
| At a bank | In Python |
|---|---|
| The queue of customers | an iterable: anything you can loop over (list, string, dict, file, range) |
| The token machine that remembers who's next | an iterator: it = iter(orders) |
| "Next, please!": one token at a time | next(it) gives the next item |
| "No more tokens": the counter closes | StopIteration tells the loop to stop |
Doing the for loop's job by hand:
orders = ["chai", "samosa"]
it = iter(orders)
print(next(it))
print(next(it))
print(next(it))
Output:
chai
samosa
Traceback (most recent call last):
File "/tmp/tmp5jy0bsj9/main.py", line 5, in <module>
print(next(it))
~~~~^^^^
StopIteration
So this loop:
for order in orders:
print(order)
is really this:
it = iter(orders)
while True:
try:
order = next(it)
except StopIteration:
break
print(order)
for calls iter() once, then next() again and again, and quietly stops when it sees StopIteration.
Iterable vs iterator
A list is the queue, not the token machine. It doesn't remember a position, so you can't call next() on it:
orders = ["chai", "samosa"]
print(next(orders))
Output:
Traceback (most recent call last):
File "/tmp/tmpsd8txhv0/main.py", line 2, in <module>
print(next(orders))
~~~~^^^^^^^^
TypeError: 'list' object is not an iterator
- Iterable: has
__iter__, soiter(x)works. You can loop over it many times; eachforgets a fresh iterator. - Iterator: has
__next__(and an__iter__that returns itself). It remembers where it is and moves forward only.
next(it, default) returns default instead of raising StopIteration when the iterator is finished.
Build your own iterator
Using dunder methods from episode 23, we can build our own token machine:
class Tokens:
def __init__(self, last):
self.last = last
self.current = 0
def __iter__(self):
return self
def __next__(self):
if self.current >= self.last:
raise StopIteration
self.current += 1
return self.current
for t in Tokens(3):
print("Token", t)
Output:
Token 1
Token 2
Token 3
__iter__returns the iterator itself.__next__gives the next token, or raisesStopIterationafter the last one.- It only ever remembers one number,
current, never a list.
It works, but it's twelve lines of class, state and StopIteration just to count to three. There must be a simpler way.
yield: the chai maker who pauses
A chai maker doesn't make a hundred cups in the morning and stack them. He makes one cup and hands it over, then waits right there, remembering how much milk is left, and when someone says "one more", carries on from exactly that spot. In Python, that word is yield.
def tokens(last):
n = 1
while n <= last:
print(" (making token", n, ")")
yield n
n += 1
gen = tokens(3)
print(type(gen))
print(next(gen))
print(next(gen))
for t in gen:
print("Token", t)
Output:
<class 'generator'>
(making token 1 )
1
(making token 2 )
2
(making token 3 )
Token 3
What happened:
- Calling
tokens(3)ran none of its code. It returned a generator object. - Each
next(gen)runs the function until the nextyield, hands out that value, and pauses, keeping all its local variables (n). - The
forloop takes whatever is left: token 3. When the function ends, the generator raisesStopIterationfor you.
Any function containing yield is a generator function, and calling it gives a generator: an iterator that Python builds for you, with __iter__, __next__ and StopIteration all written automatically. Same token machine, six lines.
Generators save memory
Back to the ten million bills. Same function, no list:
import tracemalloc
def all_bills(n):
for i in range(n):
yield i * 2 + 10
tracemalloc.start()
total = sum(all_bills(1_000_000))
print("Total:", total)
peak = tracemalloc.get_traced_memory()[1]
print(f"Peak memory: {peak:,} bytes")
Output:
Total: 1000009000000
Peak memory: 336 bytes
Same total, and a peak of a few hundred bytes, which stays the same for ten million or a billion items. sum pulls one bill, adds it, asks for the next; the pile never exists. Values made only when asked for are called lazy. (range() itself is lazy too, which is why range(10_000_000) costs almost nothing.)
Generator expressions
Remember list comprehensions from episode 16? Use round brackets instead of square ones and you get a generator expression:
import sys
squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))
print("List:", sys.getsizeof(squares_list), "bytes")
print("Generator:", sys.getsizeof(squares_gen), "bytes")
print(sum(squares_gen))
Output:
List: 8448728 bytes
Generator: 200 bytes
333332833333500000
The generator is about 200 bytes whether it will produce a million items or a billion. (sys.getsizeof of a list counts only its slots, not the numbers inside, so the real list costs even more.)
When a generator expression is the only argument of a function, you can drop its extra brackets: sum(n * n for n in range(10)).
Lazy built-ins
You've been using iterators all along: zip, map, filter, enumerate and reversed return lazy iterators, not lists.
names = ["Asha", "Ravi", "Meena"]
bills = [120, 50, 0]
pairs = zip(names, bills)
print(pairs)
print(next(pairs))
for name, amount in pairs:
print(name, amount)
print(list(map(str.upper, names)))
Output:
<zip object at 0x7f15e21abb00>
('Asha', 120)
Ravi 50
Meena 0
['ASHA', 'RAVI', 'MEENA']
zip follows the same rules as our generators: next() used up Asha, so the loop only saw Ravi and Meena. Stay lazy, and call list() only when you really need all the values.
The used-up trap
A generator makes each value once, hands it over, and forgets it. Like chai: once it's drunk, it's gone.
gen = (n * n for n in range(4))
print(list(gen))
print(list(gen))
Output:
[0, 1, 4, 9]
[]
The second list() gets nothing, with no error. This bug is silent. Also, because a generator doesn't know its items in advance:
gen = (n * n for n in range(4))
print(len(gen))
Output:
Traceback (most recent call last):
File "/tmp/tmpwx11uzz1/main.py", line 2, in <module>
print(len(gen))
~~~^^^^^
TypeError: object of type 'generator' has no len()
gen = (n * n for n in range(4))
print(gen[0])
Output:
Traceback (most recent call last):
File "/tmp/tmpcr8j9tvx/main.py", line 2, in <module>
print(gen[0])
~~~^^^
TypeError: 'generator' object is not subscriptable
If you need the values more than once, or need len() or indexing, use a list, or create a fresh generator each time.
Endless generators
Because a generator only works when asked, it can even be infinite:
def token_machine():
n = 1
while True:
yield n
n += 1
tm = token_machine()
print(next(tm), next(tm), next(tm))
print(next(tm))
Output:
1 2 3
4
It would go on forever, but it only does work when we call next(). A list could never be infinite. (Never call list() or sum() on an endless generator; take a few items with next() or itertools.islice.)
from itertools import islice
def token_machine():
n = 1
while True:
yield n
n += 1
print(list(islice(token_machine(), 5)))
Output:
[1, 2, 3, 4, 5]
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
next(it) after the last item |
StopIteration |
no items left | use a for loop, or next(it, default) |
next(["chai", "samosa"]) |
TypeError: 'list' object is not an iterator |
a list is iterable, not an iterator | next(iter(orders)) |
| using a generator twice | no error: [] the second time |
a generator is used up after one pass | a list, or a new generator |
len(gen) |
TypeError: object of type 'generator' has no len() |
the items aren't made yet | sum(1 for _ in gen) or a list |
gen[0] |
TypeError: 'generator' object is not subscriptable |
no indexing | next(gen) |
expecting tokens(3) to print |
nothing happens | calling a generator function runs no code | loop over it or call next() |
return a value instead of yield |
you get one value, not a sequence | return ends the function |
yield each value |
list(endless_generator()) |
the program never finishes / runs out of memory | it never stops | islice or next() |
Mini project: a lazy file pipeline
A pipeline is like a conveyor belt: read lines → parse → keep only chai → sum. Each stage is a generator that pulls from the stage before, so only one line travels down the belt at a time. Even a 10 GB file needs the memory of a single line. (A file object is already an iterator over its lines, from episode 20.)
with open("sales.txt", "w") as f:
f.write("chai,10\nsamosa,15\nchai,10\nvada pav,20\nchai,10\n")
def read_lines(path):
with open(path) as f:
for line in f:
yield line.strip()
def parse(lines):
for line in lines:
item, price = line.split(",")
yield item, int(price)
def only(name, sales):
for item, price in sales:
if item == name:
yield price
chai = sum(only("chai", parse(read_lines("sales.txt"))))
print("Chai sales: Rs", chai)
Output:
Chai sales: Rs 30
Nothing happens until sum starts pulling: sum asks only, only asks parse, parse asks read_lines, which reads one line from the file.
Try it: add a skip_blank stage (solution below).
Practice
1. Add a skip_blank(lines) stage to the pipeline, and test it with a file that has blank lines.
Solution
with open("sales.txt", "w") as f:
f.write("chai,10\n\nsamosa,15\n\nchai,10\n")
def read_lines(path):
with open(path) as f:
for line in f:
yield line.strip()
def skip_blank(lines):
for line in lines:
if line:
yield line
def parse(lines):
for line in lines:
item, price = line.split(",")
yield item, int(price)
print(sum(price for item, price in parse(skip_blank(read_lines("sales.txt")))))
Output:
35
Without skip_blank, the empty line would crash parse with ValueError: not enough values to unpack (expected 2, got 1).
2. A generator evens(limit) that yields even numbers from 0 up to limit.
Solution
def evens(limit):
n = 0
while n <= limit:
yield n
n += 2
print(list(evens(10)))
Output:
[0, 2, 4, 6, 8, 10]
3. An endless fibonacci() generator; print the first 10 numbers.
Solution
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fib = fibonacci()
print([next(fib) for _ in range(10)])
Output:
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
4. Rewrite the Tokens class from this episode as a class whose __iter__ is a generator (so the object can be looped over many times).
Solution
class Tokens:
def __init__(self, last):
self.last = last
def __iter__(self):
n = 1
while n <= self.last:
yield n
n += 1
t = Tokens(3)
print(list(t))
print(list(t))
Output:
[1, 2, 3]
[1, 2, 3]
Because __iter__ is a generator function, every loop gets a fresh generator: Tokens is now a reusable iterable, not a one-time iterator.
Quick quiz
1. The video's quiz: what does this print?
def f():
print("A")
yield 1
print("B")
yield 2
g = f()
print("start")
print(next(g))
Answer
Output:
start
A
1
Calling f() runs nothing. The first next() runs up to the first yield, printing A, and hands out 1. B never runs, because nobody asked for a second value.
2. What does this print?
nums = [1, 2, 3]
it = iter(nums)
print(next(it), list(it), list(nums))
Answer
Output:
1 [2, 3] [1, 2, 3]
The iterator already handed out 1, so list(it) gets the rest. The list itself is untouched and can be looped over again.
3. Which uses less memory for a billion items: [x for x in range(10**9)] or (x for x in range(10**9))?
Answer
The generator expression: it stores only its current state (about 200 bytes). The list would need many gigabytes.
Interview questions
- Iterable vs iterator? An iterable can produce an iterator (
__iter__); an iterator produces values one at a time (__next__), remembers its position, and is itself iterable (its__iter__returnsself). Lists are iterables, not iterators. - How does a
forloop work internally? It callsiter()on the object once, then callsnext()repeatedly, assigning each value to the loop variable, untilStopIterationis raised, which it catches silently. - What is a generator? An iterator created by a function containing
yield(or a generator expression). It produces values lazily and pauses between them, keeping its local state. yieldvsreturn?returnends the function and sends back one value;yieldsends back a value and pauses, resuming from the same point on the nextnext().- Why use generators? Memory efficiency (one item at a time), working with huge or infinite sequences, and building lazy pipelines where stages are composed without intermediate lists.
- What happens when you iterate a generator twice? The second pass gets nothing: generators are single-use. Re-create it, or use a list if you need multiple passes.
- Generator expression vs list comprehension?
( )creates a lazy generator;[ ]builds the full list in memory immediately. Use the generator when you only need to pass over the values once (e.g. intosum,max,any). - What is
StopIteration? The exception an iterator raises to signal it's exhausted. Inside a generator, just let the function end (orreturn); don't raise it yourself (that becomes aRuntimeErrorsince Python 3.7). - What does
yield fromdo? It delegates to another iterable, yielding each of its values:yield from other_gen()(a preview; useful for nested generators). - Name some lazy built-ins.
range,map,filter,zip,enumerate,reversed, file objects,dict.items()views, and everything initertools.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Iterable | Anything you can loop over; iter(x) works |
a list, string, dict, file, range |
| Iterator | An object that gives the next item on next() and remembers its position |
it = iter(orders) |
iter() |
Gets an iterator from an iterable | iter(["chai", "samosa"]) |
next() |
Gets the next item from an iterator | next(it) → 'chai' |
StopIteration |
The signal that an iterator has no more items | after the last next() |
| Iteration protocol | The rule: __iter__ returns an iterator, __next__ gives items, StopIteration ends |
how for works |
__iter__ |
Dunder method called by iter() |
return self |
__next__ |
Dunder method called by next() |
return self.current |
| Generator function | A function containing yield |
def tokens(last): ... yield n |
| Generator | The iterator object a generator function returns | gen = tokens(3) |
yield |
Hands out a value and pauses the function | yield n |
| Lazy evaluation | Making values only when they're asked for | generators, range |
| Eager evaluation | Making all values up front | a list comprehension |
| Generator expression | A comprehension with round brackets, giving a generator | (n * n for n in range(4)) |
| Exhausted (used up) | An iterator that has no items left | list(gen) → [] the second time |
| Infinite generator | A generator that never stops (while True) |
token_machine() |
itertools.islice |
Takes a slice of any iterator, even an endless one | islice(gen, 5) |
| Pipeline | Stages chained so each pulls from the one before | only(parse(read_lines(...))) |
| Peak memory | The most memory a program used at once | tracemalloc.get_traced_memory()[1] |
tracemalloc |
Standard-library module that measures memory Python allocates | tracemalloc.get_traced_memory()[1] |
zip, map, filter |
Built-ins that return lazy iterators | zip(names, bills) |
sys.getsizeof() |
The size of an object itself, in bytes | sys.getsizeof(gen) → 200 |
| Megabyte (MB) | About a million bytes | 40,444,808 bytes ≈ 40 MB |
TypeError |
An operation not supported by that type | len(gen), next([...]) |
Summary
- A list keeps every item in memory at once; a generator keeps only its current state.
for=iter()once, thennext()untilStopIteration.- An iterable is the queue (can be looped many times); an iterator is the token machine (remembers its place, forward only).
- An iterator class needs
__iter__and__next__, butyieldwrites all of that for you. - Calling a generator function runs no code; each
next()runs to the nextyieldand pauses. - Generator expressions use
( )instead of[ ]and cost about 200 bytes regardless of size. - A generator is used up after one pass, and has no
len()or indexing. - Chained generators form lazy pipelines that process huge files one line at a time.
Next episode: Raju wants to time every function in his billing app without editing fifty functions by hand. We invent decorators, the @ sign you already met on @property.