These are the notes for System Design episode 6. Last episode, we scaled out to many stateless servers, but we picked s1, s2 and s3 by hand. Here you invent the thing that does it for real: the load balancer. We try handing out every address through DNS (it fails), build one front door, teach it to choose with round robin and least connections, skip dead servers with health checks, meet sticky sessions, protect the balancer itself with a standby pair, and compare layer 4 and layer 7 balancers. Every Python demo below is self-contained and runs as-is; the outputs shown are real (Python 3.13). The 203.0.113.x addresses are from the range reserved for documentation (RFC 5737), not real servers.
The six problems
| Problem | What it teaches |
|---|---|
| Give users every address? | Why DNS alone isn't enough |
| One front door | The load balancer |
| Take turns | Round robin |
| Equal turns, unequal work | Least connections |
| A server dies quietly | Health checks |
| The balancer dies | A standby pair; layer 4 vs layer 7 |
Problem 1: give users every address?
ChaiNow now runs on three stateless servers, each with its own IP address. But users only know one name: chainow.in. In episode 3, DNS turned a name into an address. So which of the three should it give? The obvious idea: let DNS hand out all three, in turns, and each phone uses the one it got.
Demo 1: DNS only, one server dead
# plan A: no balancer. DNS hands out all three addresses, in turns
servers = ["203.0.113.11", "203.0.113.12", "203.0.113.13"]
alive = {"203.0.113.11": True, "203.0.113.12": False, "203.0.113.13": True}
for phone in range(1, 7):
ip = servers[(phone - 1) % 3] # this phone got this address...
status = "ok" if alive[ip] else "FAILED"
print(f"phone {phone} -> {ip} {status} (cached for 300 s)")
Output:
phone 1 -> 203.0.113.11 ok (cached for 300 s)
phone 2 -> 203.0.113.12 FAILED (cached for 300 s)
phone 3 -> 203.0.113.13 ok (cached for 300 s)
phone 4 -> 203.0.113.11 ok (cached for 300 s)
phone 5 -> 203.0.113.12 FAILED (cached for 300 s)
phone 6 -> 203.0.113.13 ok (cached for 300 s)
(phone - 1) % 3gives 0, 1, 2, 0, 1, 2: the addresses handed out in turns.- Phones 2 and 5 got the dead server
.12, and they fail. - The real problem is the cache. Remember TTL from episode 3? Those phones cached that answer, here for 300 seconds. So they keep trying the dead server for five minutes, and DNS can't take it back. One in three users is stuck.
- Why did it fail? The decision was made once, far away, and then frozen inside every phone. We need someone who decides fresh, for every single request.
Problem 2: one front door (the load balancer)
| At a hospital | In your system |
|---|---|
| One main entrance: everyone walks in here | One public address: DNS points only here |
| The token counter: "go to room 3" | Load balancer: picks a server for every request, fresh each time |
| The doctors' rooms: patients never pick | Backend servers: hidden behind the balancer; users never see them |
If one room closes, the counter just stops sending people there.
users (phones) → load balancer (one address) → s1 · s2 · s3 (servers)
- Users only ever talk to one address: the balancer's.
- It picks a server for every request.
- Servers can be added or removed behind it without users noticing.
The AWS Elastic Load Balancing docs put it simply: "A load balancer distributes incoming traffic across targets." (A target is one of the servers behind it.)
Now the real question: how should it pick?
Problem 3: take turns (round robin)
The simplest rule: take turns. s1, then s2, then s3, then back to s1.
Demo 2: round robin
from itertools import cycle
servers = ["s1", "s2", "s3"]
next_server = cycle(servers) # s1, s2, s3, s1, s2, s3, ...
count = {s: 0 for s in servers}
for request in range(9000):
count[next(next_server)] += 1
print("round robin:", count)
Output:
round robin: {'s1': 3000, 's2': 3000, 's3': 3000}
itertools.cycle(servers)repeats the list for ever;next(...)takes the next item each time.- 9,000 requests, exactly 3,000 each. Perfectly fair. This rule is called round robin.
- It's so common that nginx, a very popular load balancer, uses it by default: "When the load balancing method is not specifically configured, it defaults to round-robin."
- But fair counts aren't the same as fair work. Not every request is the same size.
Problem 4: equal turns, unequal work
A bank has three counters, and a guard sends customers to them in turns. By bad luck, every third customer has a big job (a loan application that takes 30 minutes), and they all land on counter 3. Counter 3's line keeps growing, while counters 1 and 2 finish quickly and sit idle.
Demo 3: round robin vs least connections
from itertools import cycle
# how long each request takes, in seconds: every 3rd one is a big report
jobs = [1, 1, 9] * 4
servers = ["s1", "s2", "s3"]
# round robin: take turns, no matter how busy a server is
busy = {s: 0 for s in servers}
for job, s in zip(jobs, cycle(servers)):
busy[s] += job
print("round robin :", busy)
# least connections: give the job to the server with the least work
busy = {s: 0 for s in servers}
for job in jobs:
s = min(busy, key=busy.get)
busy[s] += job
print("least connections:", busy)
Output:
round robin : {'s1': 4, 's2': 4, 's3': 36}
least connections: {'s1': 20, 's2': 13, 's3': 11}
[1, 1, 9] * 4makes 12 jobs: most take 1 second, every third is a 9-second report.zip(jobs, cycle(servers))pairs each job with the next server in turn.- Round robin:
s1ands2get 4 seconds of work each,s3gets 36. Every big report landed on the same server, so its users wait nine times longer. min(busy, key=busy.get)picks the server whose value inbusyis smallest: the least busy one.- Smarter rule: the busiest server has 20 seconds of work instead of 36.
- Real balancers measure "busy" by counting open connections, so this rule is called least connections. nginx describes it as: the "next request is assigned to the server with the least number of active connections". A connection that stays open usually means work in progress, so fewer connections is a good sign of a less busy server.
- To keep the demo short,
busyonly ever grows (jobs never finish). A real balancer's count goes down again as each request completes.
Problem 5: a server dies quietly
Back to round robin. But now server s2 has crashed, and the balancer doesn't know.
Demo 4: round robin sends users to a dead server
from itertools import cycle
servers = ["s1", "s2", "s3"]
alive = {"s1": True, "s2": False, "s3": True} # s2 has crashed
def send(server):
if not alive[server]:
raise ConnectionError(f"{server} is not responding")
return "200 OK"
for request, server in zip(range(1, 7), cycle(servers)):
try:
print(f"request {request} -> {server}: {send(server)}")
except ConnectionError as e:
print(f"request {request} -> {server}: 502 Bad Gateway ({e})")
Output:
request 1 -> s1: 200 OK
request 2 -> s2: 502 Bad Gateway (s2 is not responding)
request 3 -> s3: 200 OK
request 4 -> s1: 200 OK
request 5 -> s2: 502 Bad Gateway (s2 is not responding)
request 6 -> s3: 200 OK
sendraises aConnectionErrorif the server is down;try/exceptcatches it and shows what the user would get.- 200 OK is the HTTP status for success. 502 Bad Gateway is the status a balancer returns when the server behind it fails.
- Requests 2 and 5 hit
s2and fail. Round robin keeps faithfully sending one in three users to a dead server, because taking turns doesn't care whether a server is alive. The balancer needs to check.
The nurse who checks the rooms
| At the hospital | In the balancer |
|---|---|
| A nurse checks each room every few minutes | Health check: ask every server "are you OK?", often on a special address like /health, every few seconds |
| Doctor not answering: stop sending patients | Mark unhealthy: take it out of the list |
| Doctor back: the room reopens | Mark healthy again: put it back in |
Demo 5: health checks
from itertools import cycle
servers = ["s1", "s2", "s3"]
alive = {"s1": True, "s2": False, "s3": True} # s2 has crashed
def health_check(server): # the balancer pings /health
return alive[server]
healthy = [s for s in servers if health_check(s)]
print("healthy:", healthy)
for request, server in zip(range(1, 7), cycle(healthy)):
print(f"request {request} -> {server}: 200 OK")
Output:
healthy: ['s1', 's3']
request 1 -> s1: 200 OK
request 2 -> s3: 200 OK
request 3 -> s1: 200 OK
request 4 -> s3: 200 OK
request 5 -> s1: 200 OK
request 6 -> s3: 200 OK
- Before sending anything, the balancer checks each server and keeps only the healthy ones. Then it does round robin over the healthy list.
- A crashed server, and no user noticed.
- Real balancers do this all the time. AWS Elastic Load Balancing "monitors the health of its registered targets and ensures that it routes traffic only to healthy targets". nginx, by default, marks a server failed after 1 failed attempt (
max_fails"By default ... set to 1") and skips it for 10 seconds (fail_timeout"By default, the parameter is set to 10 seconds") before trying again.
Problem 6: sticky sessions
Some older apps aren't stateless, so they want each user to always land on the same server. That's called a sticky session. A simple way: hash the user and take it modulo (%) the number of servers. A hash turns any text into a number, always the same number for the same text. (nginx offers a method like this, called ip-hash, which uses the user's IP address.)
Demo 6: sticky by hashing, then remove a server
import zlib
users = [f"user{i}" for i in range(1, 1001)]
def pick(user, n): # sticky: hash, then mod n
return zlib.crc32(user.encode()) % n
before = {u: pick(u, 3) for u in users} # 3 servers
after = {u: pick(u, 2) for u in users} # one server removed
moved = sum(before[u] != after[u] for u in users)
print("user1 goes to server", before["user1"], "every time")
print(f"after removing a server, {moved} of 1000 users moved")
Output:
user1 goes to server 1 every time
after removing a server, 649 of 1000 users moved
zlib.crc32is a quick hash function; it needs bytes, souser.encode()turns the text into bytes first.% nturns the big number into a server number from 0 to n−1.user1goes to server 1 every time. Sticky.- But after removing one server, 649 of 1000 users (about 65%) moved. If their session lived on the old server, they're logged out.
- That's why last episode's fix, stateless servers with a shared store, is the better answer. There's also a smarter way to hash so only a few users move: consistent hashing, which gets its own episode later in the course.
Problem 7: the balancer dies
Now every request passes through one load balancer. If the token counter closes, the doctors are all fine, but nobody gets in. We just built a new single point of failure, the very thing episode 5 warned us about.
The fix: run balancers in pairs. A standby balancer watches the main one, and if it dies, the standby takes over the same public address.
Layer 4 vs layer 7
| Layer 4 · transport | Layer 7 · application | |
|---|---|---|
| Example | AWS Network Load Balancer | AWS Application Load Balancer, nginx |
| Sees | IP address + port | The whole HTTP request |
| Reads | Not the HTTP request | Path, headers, cookies |
| Speaks | TCP, UDP, TLS | HTTP, HTTPS |
| Chooses | By connection | By content, e.g. /api → API servers |
| Good at | Raw speed, any protocol | Smart routing |
| Like | A guard who only sees your ticket number | A receptionist who reads your slip |
From the AWS docs: the Network Load Balancer "Operates at the transport layer (layer 4) and supports TCP, TLS, UDP, and QUIC"; the Application Load Balancer "Operates at the application layer (layer 7) and supports HTTP and HTTPS". A layer 7 balancer can send /api to API servers and /images somewhere else.
The whole chain
- DNS hands out all IPs → cached, stuck on dead ones
- One front door → the load balancer
- Take turns → round robin
- Big jobs pile up → least connections
- Dead server, 502s → health checks
- Same server every time → sticky sessions (prefer stateless)
- Balancer dies → a standby pair
Common mistakes (and the real errors)
| Mistake | Code | Error | Fix |
|---|---|---|---|
| Every server failed its health check | healthy = [] then next(cycle(healthy)) |
StopIteration |
Alert a human, and return a clear 503 instead of crashing |
| Least connections with zero servers to pick | min({}, key=busy.get) # no servers |
ValueError: min() iterable argument is empty |
Check the server list isn't empty first |
| Hashing text instead of bytes | zlib.crc32("user1") % 3 |
TypeError: a bytes-like object is required, not 'str' |
Encode first: zlib.crc32("user1".encode()) |
The first one in full:
from itertools import cycle
healthy = []
next(cycle(healthy))
Traceback (most recent call last):
File "main.py", line 3, in <module>
next(cycle(healthy))
~~~~^^^^^^^^^^^^^^^^
StopIteration
Misconceptions
- "DNS can balance the load on its own." It can spread users out, but answers are cached, so phones stay stuck on a dead server until the TTL runs out.
- "Round robin is fair, so it's balanced." It gives equal turns, not equal work. Big jobs can pile up on one server.
- "A balancer knows when a server dies." Only if it runs health checks. Without them, it keeps sending users to the dead server (502s).
- "Sticky sessions are a good way to keep users logged in." Removing or adding a server moves most users. Stateless servers with a shared store are better.
- "Adding a load balancer removes every single point of failure." One balancer is itself a single point of failure. Run a standby pair.
Mini project: a tiny load balancer
A LoadBalancer class that combines the episode's ideas: health checks, two ways to pick (round robin or least connections), and a clear 503 when nothing is healthy.
# a tiny load balancer: health checks + round robin or least connections
from itertools import cycle
class LoadBalancer:
def __init__(self, servers):
self.servers = servers
self.alive = {s: True for s in servers}
self.busy = {s: 0 for s in servers} # seconds of work in progress
def health_check(self):
return [s for s in self.servers if self.alive[s]]
def route(self, jobs, method):
healthy = self.health_check()
if not healthy:
return "503 Service Unavailable: no healthy servers"
self.busy = {s: 0 for s in healthy}
turns = cycle(healthy)
for job in jobs:
if method == "round robin":
s = next(turns)
else: # least connections
s = min(self.busy, key=self.busy.get)
self.busy[s] += job
return self.busy
lb = LoadBalancer(["s1", "s2", "s3", "s4"])
jobs = [1, 1, 1, 9] * 3
print("round robin :", lb.route(jobs, "round robin"))
print("least connections:", lb.route(jobs, "least connections"))
lb.alive["s2"] = False # s2 crashes
print("s2 down, healthy :", lb.health_check())
print("least connections:", lb.route(jobs, "least connections"))
for s in lb.servers: # everything is down
lb.alive[s] = False
print("all down :", lb.route(jobs, "round robin"))
Output:
round robin : {'s1': 3, 's2': 3, 's3': 3, 's4': 27}
least connections: {'s1': 11, 's2': 4, 's3': 12, 's4': 9}
s2 down, healthy : ['s1', 's3', 's4']
least connections: {'s1': 10, 's3': 15, 's4': 11}
all down : 503 Service Unavailable: no healthy servers
- With four servers and every fourth job big, round robin piles all the big jobs (27 seconds) on
s4. Least connections keeps the busiest at 12. - When
s2crashes, the health check drops it and the work is shared by the other three. - With no healthy server,
routechecks first and returns a clear 503 instead of crashing withStopIterationorValueError.
Practice
- In Demo 3, change
jobsto[9, 1, 1] * 4. Predict which server round robin overloads, then run it. - In Demo 5, bring
s2back to life. What changes? - In Demo 6, change
pick(u, 2)topick(u, 4)(adding a fourth server instead of removing one). How many users move? - Round robin with health checks over
s1, s2, s3, s4;s2is unhealthy. Write code that prints where requests 1 to 5 go. - Write a
route(requests, healthy)function that returns a 503 for every request whenhealthyis empty, instead of raisingStopIteration. - Would you pick a layer 4 or a layer 7 balancer for a video call app? Why?
Solutions and hints
1. Now the big job is first in each group of three, so round robin gives every 9 to s1.
from itertools import cycle
# how long each request takes, in seconds: every 3rd one is a big report
jobs = [9, 1, 1] * 4
servers = ["s1", "s2", "s3"]
# round robin: take turns, no matter how busy a server is
busy = {s: 0 for s in servers}
for job, s in zip(jobs, cycle(servers)):
busy[s] += job
print("round robin :", busy)
# least connections: give the job to the server with the least work
busy = {s: 0 for s in servers}
for job in jobs:
s = min(busy, key=busy.get)
busy[s] += job
print("least connections:", busy)
round robin : {'s1': 36, 's2': 4, 's3': 4}
least connections: {'s1': 13, 's2': 19, 's3': 12}
Round robin's worst server has 36 seconds; least connections' has 19.
2. Change the alive line to alive = {"s1": True, "s2": True, "s3": True} # s2 is back:
healthy: ['s1', 's2', 's3']
request 1 -> s1: 200 OK
request 2 -> s2: 200 OK
request 3 -> s3: 200 OK
request 4 -> s1: 200 OK
request 5 -> s2: 200 OK
request 6 -> s3: 200 OK
s2 passes its health check again, so it goes back into the list and gets its share.
3. Change the after line to after = {u: pick(u, 4) for u in users} # one server added (and the message to "after adding a server"):
user1 goes to server 1 every time
after adding a server, 742 of 1000 users moved
Adding a server is just as bad: about three quarters of users move. Plain hash % n reshuffles almost everyone whenever n changes. That is the problem consistent hashing solves.
4.
from itertools import cycle
servers = ["s1", "s2", "s3", "s4"]
alive = {"s1": True, "s2": False, "s3": True, "s4": True}
healthy = [s for s in servers if alive[s]]
for request, server in zip(range(1, 6), cycle(healthy)):
print(f"request {request} -> {server}")
request 1 -> s1
request 2 -> s3
request 3 -> s4
request 4 -> s1
request 5 -> s3
This is the quiz below: request 4 goes to s1.
5.
from itertools import cycle
def route(requests, healthy):
if not healthy:
return ["503 Service Unavailable"] * requests
turns = cycle(healthy)
return [f"{next(turns)}: 200 OK" for _ in range(requests)]
print(route(3, ["s1", "s3"]))
print(route(3, []))
['s1: 200 OK', 's3: 200 OK', 's1: 200 OK']
['503 Service Unavailable', '503 Service Unavailable', '503 Service Unavailable']
if not healthy is true for an empty list, so we never call next on an empty cycle.
6. There's no single right answer; reason from what each layer sees and speaks. First ask: what protocol does the call traffic use? If it isn't HTTP (for example UDP), a layer 7 HTTP balancer can't read it, while a layer 4 balancer (such as AWS's Network Load Balancer, which supports TCP, TLS, UDP and QUIC) works per connection without reading the content. For the app's ordinary web and API requests (/api, /login), a layer 7 balancer is the better fit, because reading the path lets it route smartly. An app can use both.
Quick quiz
Round robin with health checks over servers s1, s2, s3, s4. s2 fails its health check. Requests 1, 2, 3 went to s1, s3, s4. Where does request 4 go?
- A. s2
- B. s1
- C. s3
- D. Nowhere: 502
Answer
B, s1. s2 is out of the healthy list, so round robin cycles s1 → s3 → s4 → s1.
Interview questions
- What is a load balancer, and why not just give users all the server addresses through DNS?
- Explain round robin and least connections. When does round robin go wrong?
- What is a health check? What happens without one? (Users get 502s from a dead server.)
- What are sticky sessions? Why are stateless servers usually better? What happens to sticky users when you add or remove a server?
- The load balancer is a single point of failure. How do you fix that? (A standby pair that takes over the same address.)
- What is the difference between a layer 4 and a layer 7 load balancer? Give an example of each.
- What does 502 Bad Gateway mean? What would you return if every server is unhealthy? (503.)
Glossary
| Term | Meaning |
|---|---|
| Load balancer | The "front door" that receives every request and picks a server to handle it |
| Backend server / target | One of the servers behind the balancer that does the real work |
| IP address | A number that identifies a machine on the network, like 203.0.113.11 |
| DNS | The internet's phone book: turns a name like chainow.in into an IP address (episode 3) |
| TTL (time to live) | How long a cached answer may be kept before it must be fetched again (episode 3) |
| Cache / cached | A saved copy of an answer, reused instead of asking again |
| Round robin | Take turns: s1, s2, s3, s1, ... |
| Least connections | Send the next request to the server with the fewest open connections (the least busy) |
| Connection | An open line of communication between a client and a server while work is in progress |
| Health check | The balancer regularly asking each server "are you OK?" and skipping those that don't answer |
/health |
A special address on a server that only answers "I'm OK"; used by health checks |
| Healthy / unhealthy | A server that passes / fails its health checks |
max_fails / fail_timeout |
nginx settings: how many failed attempts mark a server failed (default 1) and how long it is skipped (default 10 s) |
| HTTP status code | A number in a web reply that says what happened |
| 200 OK | The HTTP status for success |
| 502 Bad Gateway | The HTTP status a balancer returns when the server behind it fails |
| 503 Service Unavailable | The HTTP status for "no server can handle this right now" |
| Sticky session | Always sending the same user to the same server |
| Hash | A function that turns any text into a number, always the same number for the same text |
CRC32 (zlib.crc32) |
A quick hash function in Python's zlib module; it works on bytes |
Modulo (%) |
The remainder after division; x % 3 is always 0, 1 or 2 |
Bytes / .encode() |
Raw data a computer stores; .encode() turns text into bytes |
| Consistent hashing | A smarter way to hash so only a few users move when servers change (later episode) |
| Single point of failure | One part whose failure takes down the whole system (episode 5) |
| Standby | A spare that watches the main one and takes over if it dies |
| Layer 4 (transport layer) | The level of networking that moves data between machines; sees IP addresses and ports |
| Layer 7 (application layer) | The level of the actual request, like HTTP; sees paths, headers and cookies |
| Port | A number that says which program on a machine a connection is for |
| TCP / UDP | Transport protocols: TCP delivers data reliably and in order; UDP sends quickly without that guarantee |
| TLS | Encryption for a connection (the "S" in HTTPS) |
| QUIC | A newer transport protocol built on UDP |
| HTTP / HTTPS | The language of web requests; HTTPS is HTTP encrypted with TLS |
| Path / headers / cookies | Parts of an HTTP request: the address on the site (/api), extra information, and small saved values the browser sends back |
| nginx | A very popular web server and load balancer |
| AWS Elastic Load Balancing | Amazon's managed load balancers, including the Network (layer 4) and Application (layer 7) Load Balancers |
itertools.cycle |
Python tool that repeats a list for ever: s1, s2, s3, s1, ... |
StopIteration |
Python's signal that an iterator has nothing more to give |
ValueError / TypeError |
Python errors for a wrong value (e.g. empty input to min) / a wrong type (e.g. text instead of bytes) |
| RFC 5737 | The internet standard that reserves 203.0.113.x addresses for documentation examples |
Summary
- Handing out every address through DNS isn't enough: answers are cached, so phones stay stuck on dead servers.
- A load balancer is one front door: one public address that picks a server for every request.
- Round robin gives equal turns, not equal work. Least connections sends work to the least busy server.
- Health checks keep dead servers out of the list, so users don't get 502s.
- Sticky sessions by
hash % nmove most users when the server count changes; prefer stateless servers. - The balancer itself must not be a single point of failure: run a standby pair.
- Layer 4 balancers see IPs and ports (fast, any protocol); layer 7 balancers read the HTTP request (smart routing).
Next episode: our servers now share the load, but every one of them still asks the database the same questions, again and again. Caching: the fastest request is the one you never make.