These are the notes for Python Coding Questions, episode 19: closure and generator questions. Last episode we wrote decorators: functions that build other functions. This time we look at what those inner functions remember. Three interview questions: a counter factory (and the nonlocal keyword), lambdas made in a loop that all give the same answer, and writing your own range() as a generator with yield.
Every program was run with Python 3.13 (python3 main.py), the SyntaxPilot compiler's Python command. Every output below is real (times and memory addresses change a little from run to run).
Two words first. A scope is the region of code where a name can be used. Python looks a name up in four scopes, in this order, called LEGB: Local (the current function), Enclosing (the functions around it), Global (the file), Built-in (print, len ...).
x = "global"
def outer():
x = "enclosing"
def inner():
print(x) # not local: found in the enclosing function
inner()
outer()
print(x)
Output:
enclosing
global
Q1: A counter factory
Real-life example: a clinic has a token machine: each patient gets the next number, 1, 2, 3. A second clinic opens next door and needs its own machine, also starting at 1; the two counts must never mix. We want a factory: a function that makes token machines, where each machine keeps its own count.
Attempt 1: one global counter
count = 0
def next_token():
global count
count += 1
return count
print(next_token(), next_token()) # clinic A
print(next_token()) # clinic B starts at 3?!
Output:
1 2
3
Why it fails: there's only one count in the whole program, so every caller shares it. We need a separate count for each machine.
Attempt 2: the count inside a factory
make_counter sets up a count, defines next_token inside, and returns it:
def make_counter():
count = 0
def next_token():
count += 1
return count
return next_token
clinic_a = make_counter()
print(clinic_a())
Output:
Traceback (most recent call last):
File "main.py", line 9, in <module>
print(clinic_a())
~~~~~~~~^^
File "main.py", line 4, in next_token
count += 1
^^^^^
UnboundLocalError: cannot access local variable 'count' where it is not associated with a value
Why it fails: count += 1 means count = count + 1, an assignment. When a function assigns to a name anywhere in its body, Python treats that name as the function's own local variable for the whole function. The local count has no value yet when count + 1 reads it, so: UnboundLocalError. (Just reading an outer variable is fine; assigning is what makes it local.)
The fix: nonlocal
nonlocal count says: "count is the variable of the function around me, not a new local one".
def make_counter(start=0):
count = start
def next_token():
nonlocal count # the count from make_counter
count += 1
return count
return next_token
clinic_a = make_counter()
clinic_b = make_counter(100)
print(clinic_a(), clinic_a(), clinic_a())
print(clinic_b(), clinic_b())
print(clinic_a())
print(clinic_a.__closure__[0].cell_contents)
Output:
1 2 3
101 102
4
4
make_counterhas already returned, yet itscountstays alive, becausenext_tokenstill needs it. An inner function together with the outer variables it remembers is a closure.__closure__holds those remembered variables (Python calls each one a cell).- Every call to
make_counterruns its body again and creates a newcount, so each machine has its own state. globalis for module-level (file-level) variables;nonlocalis for the enclosing function's variables.nonlocalcan't reach a global:
count = 0
def f():
nonlocal count
count += 1
Output:
File "main.py", line 3
nonlocal count
^^^^^^^^^^^^^^
SyntaxError: no binding for nonlocal 'count' found
Complexity: each token is O(1) time; each machine stores one number, O(1) space.
A mutable object doesn't need nonlocal because you never re-assign the name, you only change the object:
def make_log():
entries = []
def log(msg):
entries.append(msg) # changes the list, no assignment to entries
return len(entries)
return log
log = make_log()
print(log("opened"), log("sold 3 chai"))
Output:
1 2
Q2: Lambdas made in a loop
A lambda is a small, one-line function with no name: lambda x: x * 2 is the same as def f(x): return x * 2.
Real-life example: a teacher hands out three cards. Each card says "multiply by the number on the board". She writes 1 on the board and hands out a card, then 2, then 3. Later the students use their cards, but the board now says 3, so every card multiplies by 3. The card never held the number, only where to look.
Attempt 1
tables = []
for i in range(1, 4):
tables.append(lambda x: x * i) # x times 1, 2, 3?
print([t(10) for t in tables])
print(i)
Output:
[30, 30, 30]
3
We expected [10, 20, 30].
Why it fails
A closure remembers the variable i, not its value at the moment the lambda is made. It looks up i when the lambda is called. By then the loop has finished and i is 3. This is called late binding. All three lambdas share the same i:
tables = [lambda x: x * i for i in range(1, 4)]
print([t(10) for t in tables])
print(tables[0].__closure__[0] is tables[2].__closure__[0])
Output:
[30, 30, 30]
True
The fix: save the value when the function is made
Episode 18's rule: a default value is evaluated once, when the function is defined. Last time that was a bug; here it's exactly what we want. i=i makes a parameter i whose default is the loop's current i, saved right now. Or use functools.partial, which stores argument values inside a new function:
from functools import partial
tables = []
for i in range(1, 4):
tables.append(lambda x, i=i: x * i) # i saved NOW
print([t(10) for t in tables])
def times(x, n):
return x * n
tables = [partial(times, n=i) for i in range(1, 4)]
print([t(10) for t in tables])
Output:
[10, 20, 30]
[10, 20, 30]
A third fix is a factory function (a fresh enclosing scope for every value):
def make_times(n):
return lambda x: x * n # each call has its own n
tables = [make_times(i) for i in range(1, 4)]
print([t(10) for t in tables])
Output:
[10, 20, 30]
You'll meet this bug wherever functions are created in a loop: GUI buttons, callbacks, threads. Calling each lambda is O(1).
Q3: Write your own range()
Real-life example: at a dinner, one cook makes 1,000 rotis before anyone eats: the guests wait, and the pile fills the kitchen. Another cook makes one roti when a plate is empty: always hot, no pile. When a guest asks for the next one, she carries on from where she stopped. In Python, that second cook is a generator; handing over one roti is yield; asking for the next is next().
Attempt 1: build a list
import sys, time
def my_range(stop):
out = []
i = 0
while i < stop:
out.append(i)
i += 1
return out
start = time.perf_counter()
nums = my_range(5_000_000)
print("first value:", nums[0])
print(f"waited {time.perf_counter() - start:.2f} s for it")
print(sys.getsizeof(nums), "bytes")
Output:
first value: 0
waited 0.31 s for it
43947864 bytes
We wanted one number and still waited for all five million to be made, and the list alone takes about 44 MB (sys.getsizeof gives an object's own size in bytes; the int objects inside take even more). A trillion numbers would need terabytes. But Python's own range is instant at any size:
import sys
r = range(10**12)
print(sys.getsizeof(r), "bytes")
print(r[0], r[1], len(r))
Output:
48 bytes
0 1 1000000000000
yield: hand out one value, then pause
A function with yield in its body is a generator function. Calling it doesn't run the body: it returns a generator object, a paused function. Each next(g) runs it up to the next yield, hands out that value, and freezes there with all its local variables. The next next(g) continues from that line. When the function ends, next raises StopIteration.
def my_range(stop):
i = 0
while i < stop:
yield i # hand out one value, pause here
i += 1
g = my_range(3)
print(g)
print(next(g), next(g), next(g))
print(next(g))
Output:
<generator object my_range at 0x7f67ddeed9c0>
0 1 2
Traceback (most recent call last):
File "main.py", line 10, in <module>
print(next(g))
~~~~^^^
StopIteration
A for loop calls next() for you and stops quietly at StopIteration. You can watch the pausing:
def cook(n):
for k in range(1, n + 1):
print(f" (cooking roti {k})")
yield f"roti {k}"
print(" (kitchen closed)")
for roti in cook(2):
print("eating", roti)
Output:
(cooking roti 1)
eating roti 1
(cooking roti 2)
eating roti 2
(kitchen closed)
Attempt 2: add start and step
def my_range(start, stop, step=1):
i = start
while i < stop:
yield i
i += step
print(list(my_range(2, 10, 3)))
print(list(my_range(10, 0, -3)))
print(list(range(10, 0, -3)))
Output:
[2, 5, 8]
[]
[10, 7, 4, 1]
Why it fails: with a negative step we count down, so the loop should run while i > stop. 10 < 0 is false, so the loop never starts and we get [].
The fix: the check depends on the step's sign
import sys
def my_range(start, stop=None, step=1):
if stop is None: # my_range(5) means 0 to 5
start, stop = 0, start
if step == 0:
raise ValueError("step must not be zero")
i = start
while (step > 0 and i < stop) or (step < 0 and i > stop):
yield i
i += step
print(list(my_range(5)))
print(list(my_range(2, 10, 3)))
print(list(my_range(10, 0, -3)))
print(sys.getsizeof(my_range(10**12)), "bytes")
big = my_range(10**12)
print(next(big), next(big))
g = my_range(4)
print(sum(g), sum(g))
Output:
[0, 1, 2, 3, 4]
[2, 5, 8]
[10, 7, 4, 1]
216 bytes
0 1
6 0
- One argument means
stop, starting from 0, like the realrange. - A step of 0 would never reach
stop, so raiseValueError(the realrangedoes too). - A trillion-number generator is 216 bytes: only
i,stop,stepand the paused position are kept. O(1) memory, O(1) time per value. - A generator can be walked once. The first
sum(g)used it up, so the second gives 0. Make a new generator, or keep a list if you need the values twice.
Our version still differs from the real range in two ways worth knowing: range is a sequence (you can index it, take len(), and loop over it many times), while a generator only goes forward, once.
print(range(10, 0, -3)[1], len(range(10, 0, -3)))
def gen():
yield 1
print(len(gen()))
Output:
7 4
Traceback (most recent call last):
File "main.py", line 4, in <module>
print(len(gen()))
~~~^^^^^^^
TypeError: object of type 'generator' has no len()
Common mistakes (and what you really see)
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| one global counter | clinic B's first token is 3 | one count for everyone | a factory that makes a closure |
count += 1 in the inner function |
UnboundLocalError: cannot access local variable 'count' where it is not associated with a value |
assigning makes the name local | nonlocal count |
nonlocal for a module variable |
SyntaxError: no binding for nonlocal 'count' found |
nonlocal only sees enclosing functions |
global count (or better, a closure) |
lambda x: x * i in a loop |
[30, 30, 30] |
i is read at call time |
lambda x, i=i: x * i, partial, or a factory |
| returning a list of n values | 44 MB and a wait for 5,000,000 | the whole pile first | yield one at a time |
print(g) |
<generator object my_range at 0x...> |
a generator is not a list | list(g) or loop over it |
next(g) after the end |
StopIteration |
no values left | use a for loop, or next(g, default) |
while i < stop with a negative step |
[] for my_range(10, 0, -3) |
counting down | i > stop when step < 0 |
sum(g) twice |
6, then 0 |
a generator is used up | make a new one, or a list |
len(g) |
TypeError: object of type 'generator' has no len() |
a generator doesn't know its length | len(list(g)) or count while looping |
next(g, default) returns the default instead of raising at the end:
g = iter([7])
print(next(g, "done"), next(g, "done"))
Output:
7 done
Your turn: a running average (solution)
Question: write make_averager(). Each call adds a new mark and returns the average of all marks so far: 10.0, 15.0, 30.0.
Idea: a closure that keeps a running total and a count, changed with nonlocal. Keeping two numbers (not the whole list) makes each call O(1) time and O(1) space.
def make_averager():
total = 0
count = 0
def add(mark):
nonlocal total, count
total += mark
count += 1
return total / count
return add
avg = make_averager()
print(avg(10), avg(20), avg(60))
other = make_averager()
print(other(99))
Output:
10.0 15.0 30.0
99.0
Bonus: chunks(items, size), a generator that hands out a list in pieces of size items (the last piece may be shorter):
def chunks(items, size):
if size <= 0:
raise ValueError("size must be positive")
for start in range(0, len(items), size):
yield items[start:start + size]
print(list(chunks([1, 2, 3, 4, 5, 6, 7], 3)))
print(list(chunks([], 3)))
Output:
[[1, 2, 3], [4, 5, 6], [7]]
[]
Mini project: a token display board
A closure counter per counter desk, and a generator that merges their calls in turn, the way a bank's display board shows "Token A1, B101, A2, ...".
def make_desk(prefix, start=1):
n = start - 1
def call():
nonlocal n
n += 1
return f"{prefix}{n}"
return call
def board(desks, rounds):
for _ in range(rounds):
for desk in desks:
yield desk()
cash = make_desk("A")
loans = make_desk("B", 101)
for token in board([cash, loans], 3):
print("Now serving:", token)
print("next cash token:", cash())
Output:
Now serving: A1
Now serving: B101
Now serving: A2
Now serving: B102
Now serving: A3
Now serving: B103
next cash token: A4
Practice questions (with solutions)
1. Write an endless fibonacci() generator and take the first 10 values.
Solution
An endless generator never returns; the caller decides when to stop. itertools.islice takes the first n values from any iterator.
from itertools import islice
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
print(list(islice(fibonacci(), 10)))
g = fibonacci()
print([next(g) for _ in range(5)])
Output:
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
[0, 1, 1, 2, 3]
2. Write countdown(n) as a generator: n, n − 1, ..., 1, then "Go!".
Solution
def countdown(n):
while n > 0:
yield n
n -= 1
yield "Go!"
print(list(countdown(3)))
print(list(countdown(0)))
Output:
[3, 2, 1, 'Go!']
['Go!']
3. What does this print, and how do you fix it?
buttons = {}
for name in ["chai", "coffee"]:
buttons[name] = lambda: f"You ordered {name}"
print(buttons["chai"]())
Solution
Late binding: both lambdas read name when called, after the loop left it at "coffee". Save it with a default.
buttons = {}
for name in ["chai", "coffee"]:
buttons[name] = lambda: f"You ordered {name}"
print(buttons["chai"]())
for name in ["chai", "coffee"]:
buttons[name] = lambda name=name: f"You ordered {name}"
print(buttons["chai"]())
Output:
You ordered coffee
You ordered chai
4. A generator expression: the sum of the squares of 1 to 10 million, in O(1) memory.
Solution
(x * x for x in ...) with round brackets is a generator expression: like a list comprehension, but lazy (values made one at a time).
import sys
gen = (x * x for x in range(1, 10_000_001))
print(sys.getsizeof(gen), "bytes")
print(sum(gen))
Output:
200 bytes
333333383333335000000
5. Write make_limited(func, n): a closure that lets func run at most n times, then returns "limit reached".
Solution
def make_limited(func, n):
left = n
def wrapper(*args, **kwargs):
nonlocal left
if left == 0:
return "limit reached"
left -= 1
return func(*args, **kwargs)
return wrapper
send_otp = make_limited(lambda phone: f"OTP sent to {phone}", 2)
print(send_otp("98xxxx01"))
print(send_otp("98xxxx01"))
print(send_otp("98xxxx01"))
Output:
OTP sent to 98xxxx01
OTP sent to 98xxxx01
limit reached
Quick quiz
What does this print?
def squares(n):
for i in range(n):
yield i * i
g = squares(4)
print(next(g), list(g), list(g))
(A) 0 [0, 1, 4, 9] [0, 1, 4, 9] (B) 0 [1, 4, 9] [] (C) 0 [1, 4, 9] [1, 4, 9] (D) [0, 1, 4, 9] [] []
Answer
(B). next(g) takes the first value, 0. The first list(g) takes everything left: [1, 4, 9]. The generator is now used up, so the second list(g) is [].
def squares(n):
for i in range(n):
yield i * i
g = squares(4)
print(next(g), list(g), list(g))
Output:
0 [1, 4, 9] []
Interview questions
- What is a closure? An inner function together with the variables from its enclosing function that it uses. They stay alive after the outer function returns (
f.__closure__). - Why does
count += 1inside an inner function raiseUnboundLocalError? Assignment makescountlocal to the inner function for its whole body; it's read before it has a value. Declarenonlocal count. globalvsnonlocal?globalrefers to a module-level name;nonlocalto a name in the nearest enclosing function.- What is LEGB? Python's name lookup order: Local, Enclosing, Global, Built-in.
- Why do lambdas created in a loop all return the same value? Late binding: closures look up variables at call time, and the loop variable has its final value by then. Fix with a default argument (
i=i),functools.partial, or a factory function. - What is a generator? A function with
yield; calling it returns an iterator that produces values lazily, pausing at eachyieldand resuming onnext(). - Generator vs list? A generator uses O(1) memory and produces values on demand, but can be walked only once and has no
len()or indexing. A list holds everything in memory. - What happens when a generator finishes?
next()raisesStopIteration;forloops handle it for you. - What's a generator expression?
(expr for x in iterable): a lazy version of a list comprehension. - How would you implement
rangeyourself? A generator takingstart, stop=None, step=1; handle one argument, rejectstep == 0, and loop whilei < stopfor positive steps ori > stopfor negative steps.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Scope | Where in the code a name can be used | a function's local names |
| LEGB | Lookup order: Local, Enclosing, Global, Built-in | print is built-in |
| Local variable | A name assigned inside a function | count = 0 in make_counter |
| Enclosing function | The function around an inner function | make_counter around next_token |
| Global variable | A name at module (file) level | count = 0 at the top |
global |
Assign to a module-level name from a function | global count |
nonlocal |
Assign to the enclosing function's name | nonlocal count |
UnboundLocalError |
A local name read before it has a value | count += 1 without nonlocal |
| Factory function | A function that builds and returns new objects or functions | make_counter() |
| Closure | An inner function plus the outer variables it remembers | clinic_a |
Cell / __closure__ |
Where a closure keeps a remembered variable | f.__closure__[0].cell_contents |
| Lambda | A one-line function with no name | lambda x: x * 2 |
| Late binding | A closure reads a variable's value when called, not when made | [30, 30, 30] |
functools.partial |
Makes a new function with some arguments fixed | partial(times, n=2) |
| Generator function | A function containing yield |
def my_range(stop): ... yield i |
| Generator | The paused-function object it returns | g = my_range(3) |
yield |
Hand out one value and pause | yield i |
next() |
Run a generator to its next yield |
next(g) |
StopIteration |
Raised by next() when no values are left |
after the last value |
| Iterator | Anything next() works on, used up as you go |
a generator, iter([1, 2]) |
| Lazy | Values made only when asked for | generators, range |
| Generator expression | A lazy comprehension in round brackets | (x * x for x in nums) |
sys.getsizeof |
An object's own size in bytes | sys.getsizeof([1, 2]) |
itertools.islice |
Take a slice of any iterator | islice(fibonacci(), 10) |
| Sequence | Indexable, sized, reusable (list, tuple, range) | range(5)[2] |
Summary
- A factory function gives each object its own state; a global counter is shared by everyone.
- Assigning to a name makes it local. To change an enclosing function's variable, declare
nonlocal. - A closure is an inner function plus the variables it remembers; they live on after the outer function returns.
- Closures remember variables, not values (late binding). Lambdas made in a loop all see the final loop value; save it with
i=i,partial, or a factory. yieldmakes a generator: one value at a time, pausing in between. O(1) memory, any size.next()ends withStopIteration.- Write
my_rangewith the loop check chosen by the step's sign, rejectstep == 0, and remember a generator is used up after one pass. - Next episode: OOP questions:
__eq__with__hash__, aStackclass with__len__and__repr__, and the class attribute trap.