SyntaxPilot
Courses/Python/Lesson 26

Python · Lesson 26 of 40 · 10 min

Context managers and with

queued

These are the notes for Python episode 26. Since episode 20 you've been writing with open(...) as f:. In this episode you find out what with really does: a bad price crashes Raju's save halfway, and the file ends up empty and still open, because the clean-up line never ran. We fix it with try/finally, then invent context managers: objects with __enter__ (setup) and __exit__ (clean-up that always runs). We cover what __exit__ receives when there's an error, three traps (using a file after the block, objects that don't support with, and __exit__ swallowing errors), the @contextmanager shortcut with yield, suppress(), managing two files in one with, and a mini project: a stock transaction that rolls back. Every example below runs as-is in the SyntaxPilot compiler, with its real output from Python 3.13.

The problem: a crash empties the file

Every morning Raju opens the shop shutter, and every night he pulls it down. A file is the same: open, use, close. Then one day there's a power cut at noon and everyone runs out: the shutter is left open. In code, an error is that power cut.

def save_sales(sales):
    f = open("sales.txt", "w")
    for item, price in sales:
        f.write(f"{item},{int(price)}\n")
    f.close()

try:
    save_sales([("chai", "10"), ("samosa", "ten")])
except ValueError as e:
    print("Error:", e)
    print("File says:", repr(open("sales.txt").read()))

Output:

Error: invalid literal for int() with base 10: 'ten'
File says: ''

int("ten") fails (a ValueError, episode 18), which is fair. But the file is empty, even though chai,10 was written before the crash. (repr() shows the text with its quotes, so an empty string appears as ''.)

Why it fails: Python doesn't write to the disk on every write(). It collects text in a buffer, a waiting area in memory, and saves it when the buffer fills up or the file is closed. The error jumped out of the loop, so f.close() never ran: the file is still open and chai,10 is still sitting in the buffer. In a long-running billing app, files left open pile up and data is lost. Clean-up must run no matter what.

The old fix: try / finally

From episode 18: code in finally runs whether the try block finished normally or raised an error.

f = open("sales.txt", "w")
try:
    f.write("chai,10\n")
    price = int("ten")
finally:
    f.close()
    print("closed?", f.closed)

Output:

closed? True
Traceback (most recent call last):
  File "main.py", line 4, in <module>
    price = int("ten")
ValueError: invalid literal for int() with base 10: 'ten'

The file was closed, and then the error was still reported. It works, but every open now needs a try, a finally and a close, in every function. Forget one and the bug is back. We'd rather package the setup and the clean-up together, once.

Inventing context managers: __enter__ and __exit__

Think of a hotel. At check-in you get a key card (setup). Then your stay (whatever you do in the room). At check-out the card stops working, whether you had a great stay or the AC broke (clean-up, always).

At a hotel In Python
Check in, get the key card __enter__() runs at the start of with; its return value goes into as
Your stay the indented block under with
Check out, the card stops working __exit__() runs when the block ends, even after an error
class Shift:
    def __init__(self, cashier):
        self.cashier = cashier

    def __enter__(self):
        print(f"{self.cashier} opens the drawer")
        return self

    def __exit__(self, exc_type, exc, tb):
        print(f"{self.cashier} locks the drawer")
        print("  error was:", exc_type)

with Shift("Ravi") as s:
    print("selling chai...")

with Shift("Asha") as s:
    print("selling samosa...")
    price = int("ten")

Output:

Ravi opens the drawer
selling chai...
Ravi locks the drawer
  error was: None
Asha opens the drawer
selling samosa...
Asha locks the drawer
  error was: <class 'ValueError'>
Traceback (most recent call last):
  File "main.py", line 18, in <module>
    price = int("ten")
ValueError: invalid literal for int() with base 10: 'ten'
  • __enter__ returns self, so as s receives the Shift object.
  • __exit__ gets three extra arguments about any error: its type (exc_type), the error object (exc) and the traceback (tb). With no error, all three are None.
  • Asha's shift crashed, but the drawer was still locked, and then the error carried on as normal.

Any object with __enter__ and __exit__ is a context manager, and with is the statement that uses one. You just invented context managers.

What with open(...) does

A file object is a context manager too.

with open("sales.txt", "w") as f:
    f.write(...)
  1. open(...) makes the file object (the context manager).
  2. Python calls its __enter__(); the return value (the file itself) goes into f.
  3. Your block runs.
  4. When the block ends, normally or because of an error, __exit__() runs: it closes the file, which also writes the buffer to the disk.

Roughly, with is this try/finally written for you:

manager = open("sales.txt", "w")
f = manager.__enter__()
try:
    f.write(...)
finally:
    manager.__exit__(...)   # with the error details, if there was one

The same save, with with:

def save_sales(sales):
    with open("sales.txt", "w") as f:
        for item, price in sales:
            f.write(f"{item},{int(price)}\n")
    print("closed?", f.closed)

try:
    save_sales([("chai", "10"), ("samosa", "ten")])
except ValueError as e:
    print("Error:", e)
    print("File says:", repr(open("sales.txt").read()))

save_sales([("chai", "10"), ("samosa", "15")])

Output:

Error: invalid literal for int() with base 10: 'ten'
File says: 'chai,10\n'
closed? True

Same error, but this time chai,10 was saved, because the file was closed on the way out.

Three traps

Trap 1: using the file after the block

with open("sales.txt", "w") as f:
    f.write("chai,10\n")

f.write("samosa,15\n")

Output:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    f.write("samosa,15\n")
    ~~~~~~~^^^^^^^^^^^^^^^
ValueError: I/O operation on closed file.

The variable f still exists, but the file was closed when the block ended. (I/O = input/output.) Do all file work inside the block.

Trap 2: an object that isn't a context manager

class Shop:
    pass

with Shop() as s:
    print("hi")

Output:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    with Shop() as s:
         ~~~~^^
TypeError: 'Shop' object does not support the context manager protocol

A protocol here means the agreed set of methods: __enter__ and __exit__. Shop has neither.

Trap 3: __exit__ returning True swallows the error

If __exit__ returns a true value, Python treats the error as handled and hides it.

class Quiet:
    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc, tb):
        print("cleaning up")
        return True

with Quiet():
    total = 100 / 0
    print("this never runs")

print("total is", total)

Output:

cleaning up
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    print("total is", total)
                      ^^^^^
NameError: name 'total' is not defined

No ZeroDivisionError at all: it was swallowed. The program fails later with a confusing NameError, because total was never set. Return True only when you really mean to ignore the error; normally return nothing (None).

The shortcut: @contextmanager

Writing a class every time is a lot. The contextlib module ("tools for context managers") has the contextmanager decorator (episode 25), used on a generator (episode 24):

  • everything before yield is the setup (__enter__),
  • the value you yield goes into as,
  • everything after yield is the clean-up (__exit__); put it in finally so it runs even on an error (the error comes out of the function at the yield line).
from contextlib import contextmanager

@contextmanager
def shift(cashier):
    print(f"{cashier} opens the drawer")
    try:
        yield cashier.upper()
    finally:
        print(f"{cashier} locks the drawer")

with shift("Ravi") as name:
    print("serving as", name)

with shift("Asha"):
    int("ten")

Output:

Ravi opens the drawer
serving as RAVI
Ravi locks the drawer
Asha opens the drawer
Asha locks the drawer
Traceback (most recent call last):
  File "main.py", line 15, in <module>
    int("ten")
    ~~~^^^^^^^
ValueError: invalid literal for int() with base 10: 'ten'

A whole context manager in seven lines. Without the try/finally, Asha's drawer would not be locked.

Handy built-ins

import time
from contextlib import contextmanager, suppress

@contextmanager
def timer(label):
    start = time.perf_counter()
    yield
    print(f"{label} took {time.perf_counter() - start:.1f}s")

with timer("closing report"):
    time.sleep(0.3)

with suppress(FileNotFoundError):
    open("old_bills.txt")
print("no old bills, carrying on")

Output:

closing report took 0.3s
no old bills, carrying on
  • timer is a context manager we wrote; a plain yield (no value) is fine when as isn't needed.
  • suppress(FileNotFoundError) ignores exactly that one kind of error inside its block. It's the safe version of trap 3: you name exactly which error to ignore.

Two files in one with

with open("in.txt", "w") as f:
    f.write("chai\nsamosa\n")

with open("in.txt") as src, open("out.txt", "w") as dst:
    for line in src:
        dst.write(line.upper())

print(open("out.txt").read())

Output:

CHAI
SAMOSA

Separate the managers with a comma. Both files are closed at the end, in reverse order (dst, then src). Thread locks (episode 31), database connections and network connections all work with with too.

Common mistakes

Mistake What you see Why Fix
f = open(...), crash before f.close() no error: the file is empty ('') and still open close() was skipped; the buffer was never written with open(...) as f:
f.write(...) after the with block ValueError: I/O operation on closed file. with already closed it do the work inside the block
with Shop() as s: TypeError: 'Shop' object does not support the context manager protocol no __enter__ / __exit__ add both, or use @contextmanager
__exit__ returns True no error: the exception is swallowed (later NameErrors) True means "handled" return None, or use suppress(SpecificError)
yield without try/finally in @contextmanager clean-up skipped when the block raises the error comes out at yield try: yield ... finally: clean up
forgetting return self in __enter__ as s gives None (AttributeError: 'NoneType' object has no attribute ...) __enter__ returned nothing return self
__exit__(self) with no extra parameters TypeError: Shift.__exit__() takes 1 positional argument but 4 were given Python always passes three error arguments def __exit__(self, exc_type, exc, tb):

Mini project: stock rollback

An order takes milk, tea and sugar from the stock. If the sugar runs out halfway, the milk already taken must go back: all or nothing. Banks and databases call this a transaction; undoing it is a rollback. Setup copies the stock; if anything fails, clean-up puts the copy back and re-raises the error so it isn't hidden.

from contextlib import contextmanager

stock = {"milk": 5, "tea": 10, "sugar": 3}

@contextmanager
def transaction(store):
    backup = store.copy()
    try:
        yield store
    except Exception as e:
        store.clear()
        store.update(backup)
        print("Rolled back:", e)
        raise

def sell(store, item, qty):
    if store[item] < qty:
        raise ValueError(f"only {store[item]} {item} left")
    store[item] -= qty

with transaction(stock):
    sell(stock, "milk", 2)
    sell(stock, "tea", 2)
print("After order 1:", stock)

try:
    with transaction(stock):
        sell(stock, "milk", 2)
        sell(stock, "sugar", 5)
except ValueError:
    pass
print("After order 2:", stock)

Output:

After order 1: {'milk': 3, 'tea': 8, 'sugar': 3}
Rolled back: only 3 sugar left
After order 2: {'milk': 3, 'tea': 8, 'sugar': 3}

Order 2 took 2 milk, then failed on sugar, and the milk came back: the stock is exactly as before. store.clear() + store.update(backup) restores the same dictionary object (so stock outside sees it); a bare raise inside except re-raises the current error.

Practice

1. A Shift class that counts sales and prints the count on exit.

Solution
class Shift:
    def __init__(self, cashier):
        self.cashier = cashier
        self.sales = 0

    def __enter__(self):
        print(f"{self.cashier} starts")
        return self

    def sell(self, item):
        self.sales += 1
        print("  sold", item)

    def __exit__(self, exc_type, exc, tb):
        print(f"{self.cashier} ends after {self.sales} sales")

with Shift("Ravi") as s:
    s.sell("chai")
    s.sell("samosa")

Output:

Ravi starts
  sold chai
  sold samosa
Ravi ends after 2 sales

2. A @contextmanager called indent() so that nested blocks print further to the right.

Solution
from contextlib import contextmanager

level = 0

@contextmanager
def indent():
    global level
    level += 1
    try:
        yield
    finally:
        level -= 1

def say(text):
    print("    " * level + text)

say("Menu")
with indent():
    say("Drinks")
    with indent():
        say("chai")
        say("coffee")
    say("Snacks")
say("End")

Output:

Menu
    Drinks
        chai
        coffee
    Snacks
End

3. Copy a file into a backup file using a single with.

Solution
with open("bills.txt", "w") as f:
    f.write("chai,10\nsamosa,15\n")

with open("bills.txt") as src, open("backup.txt", "w") as dst:
    for line in src:
        dst.write(line)

with open("backup.txt") as f:
    print(f.read())

Output:

chai,10
samosa,15

4. Run several orders through the rollback project, and print the stock inside the failing order before the rollback.

Solution
from contextlib import contextmanager

stock = {"milk": 5, "tea": 10, "sugar": 3}

@contextmanager
def transaction(store):
    backup = store.copy()
    try:
        yield store
    except Exception as e:
        print("  inside, before rollback:", store)
        store.clear()
        store.update(backup)
        print("Rolled back:", e)
        raise

def sell(store, item, qty):
    if store[item] < qty:
        raise ValueError(f"only {store[item]} {item} left")
    store[item] -= qty

for order in [[("milk", 2)], [("tea", 4), ("sugar", 9)], [("sugar", 3)]]:
    try:
        with transaction(stock):
            for item, qty in order:
                sell(stock, item, qty)
    except ValueError:
        pass
    print("Stock:", stock)

Output:

Stock: {'milk': 3, 'tea': 10, 'sugar': 3}
  inside, before rollback: {'milk': 3, 'tea': 6, 'sugar': 3}
Rolled back: only 3 sugar left
Stock: {'milk': 3, 'tea': 10, 'sugar': 3}
Stock: {'milk': 3, 'tea': 10, 'sugar': 0}

Quick quiz

1. The video's quiz: what does this print?

class Door:
    def __enter__(self):
        print("open")
    def __exit__(self, *err):
        print("close")

with Door():
    print("inside")
print("after")
Answer
open
inside
close
after

__enter__ runs first, then the block, then __exit__ as soon as the block ends, and only then the next line. (*err collects the three error arguments.)

2. What does this print?

from contextlib import contextmanager

@contextmanager
def tag(name):
    print(f"<{name}>")
    yield
    print(f"</{name}>")

with tag("b"):
    print("chai")
Answer
<b>
chai
</b>

3. If the block inside with open(...) as f: raises an error, is the file closed?

Answer

Yes. __exit__ always runs, so the file is closed and the buffer is written; then the error carries on (a file's __exit__ doesn't swallow it).

Interview questions

  1. What is a context manager? An object that defines __enter__ and __exit__, used with with to guarantee setup and clean-up around a block of code.
  2. What does the with statement do? Calls __enter__, binds its result to the as name, runs the block, then always calls __exit__ with the exception details (or Nones), like a built-in try/finally.
  3. What are __exit__'s parameters? exc_type, exc_value, traceback: all None if the block finished normally.
  4. How can a context manager suppress an exception? By returning a true value from __exit__. Use this deliberately and narrowly, e.g. contextlib.suppress(FileNotFoundError).
  5. How does @contextmanager work? It turns a generator into a context manager: code before yield is setup, the yielded value is bound by as, and code after yield is clean-up. Exceptions in the block are re-raised at the yield, so wrap it in try/finally.
  6. Why is with open(...) better than open() + close()? The file is always closed (and flushed) even if an error occurs, with less code and no way to forget.
  7. Name some real context managers. Files, threading.Lock, database connections/transactions, decimal.localcontext(), tempfile.TemporaryDirectory(), unittest.mock.patch, contextlib.suppress, contextlib.redirect_stdout.
  8. Can one with manage several resources? Yes: with a() as x, b() as y:; they are exited in reverse order. For a variable number, use contextlib.ExitStack.
  9. What is a transaction? A group of changes that happens completely or not at all; on failure, the changes are rolled back.

Glossary

Term Meaning Example
Context manager An object with __enter__ and __exit__, used with with a file object
with statement Runs setup, the block, then always the clean-up with open("a.txt") as f:
__enter__ Setup; its return value goes into as return self
__exit__ Clean-up; always runs when the block ends def __exit__(self, exc_type, exc, tb):
as Names the value __enter__ returned as f
Clean-up Code that releases a resource: close, unlock, disconnect f.close()
Resource Something that must be given back: a file, lock, connection an open file
Buffer A waiting area in memory where written text sits until it's saved to disk text before close()
try / finally finally runs whether or not there was an error finally: f.close()
f.closed True if the file has been closed print(f.closed)
exc_type, exc, tb The error's type, the error object, the traceback (None if no error) <class 'ValueError'>
Traceback The report of where an error happened Traceback (most recent call last):
Context manager protocol The rule that an object must have __enter__ and __exit__ 'Shop' object does not support...
Swallowing an error Hiding an exception so it's never reported __exit__ returning True
contextlib Standard-library module of context-manager tools from contextlib import contextmanager
@contextmanager Turns a generator (setup, yield, clean-up) into a context manager @contextmanager def timer(...)
suppress() Ignores one named kind of error inside its block with suppress(FileNotFoundError):
FileNotFoundError The error for opening a file that doesn't exist open("old_bills.txt")
I/O Input/output: reading and writing files, the screen, the network I/O operation on closed file.
repr() Shows a value as code, e.g. text with its quotes repr("") → "''"
Transaction A group of changes that happens completely or not at all an order
Rollback Undoing a failed transaction's changes restoring the stock backup
Re-raise Raising the same error again after handling it partly bare raise in except

Summary

  • An error can skip f.close(): the file stays open and the buffer is lost ('').
  • try/finally fixes it but is easy to forget; a context manager packages setup and clean-up together.
  • __enter__ sets up (its return value goes into as); __exit__ cleans up and always runs, receiving the error details.
  • with open(...) closes the file for you, even after a crash.
  • Traps: using a file after the block, objects without the protocol, and __exit__ returning True (swallowed errors).
  • @contextmanager: setup, yield, clean-up in finally.
  • suppress(SpecificError) ignores one named error; one with can manage several resources.
  • A context manager can make changes all or nothing (a transaction with rollback).

Next episode: a price arrives as text and two samosas cost 1,515 rupees. We learn type hints, a type checker, and dataclasses that write __init__ for you.