SyntaxPilot
Courses/System Design/Lesson 5

System Design · Lesson 5 of 40 · 11 min

Scaling: vertical vs horizontal

queued

These are the notes for System Design episode 5. Last episode, we saw that past your throughput limit a queue forms and latency explodes. Here we fix that for real. Your app gets popular, one machine isn't enough, and you invent both kinds of scaling yourself: capacity, vertical scaling (a bigger machine) and its limits, the single point of failure, horizontal scaling (more machines), and why servers must be stateless with a shared store. Every Python demo below is self-contained and runs as-is; the outputs shown are real (Python 3.13). The 100 requests per second per server and 25 per CPU core are teaching numbers, not measurements of a real machine.

The six problems

Problem What it teaches
Traffic doubles overnight Capacity: past it, requests fail
"Just buy a bigger server" Vertical scaling (scale up)
The big server dies at 2 AM Single point of failure
Many small servers Horizontal scaling (scale out)
"Why am I logged out?" State stuck in one server's memory
One shared store Stateless servers

Problem 1: your chai app goes viral (capacity)

You build a little app called ChaiNow: people order chai from their phones, and it all runs on one server. Then someone posts a reel about it and it goes viral. Traffic doubles, then doubles again, and orders start failing: users see "something went wrong". Nothing in your code changed. So what broke?

Every server has a capacity: the most requests per second it can handle. Say ours handles 100.

Demo 1: traffic past capacity

# one server can handle 100 requests per second
CAPACITY = 100

for traffic in [50, 100, 200, 400]:
    served = min(traffic, CAPACITY)
    failed = traffic - served
    print(f"{traffic:>3} req/s -> served {served:>3}, failed {failed:>3}")

Output:

 50 req/s -> served  50, failed   0
100 req/s -> served 100, failed   0
200 req/s -> served 100, failed 100
400 req/s -> served 100, failed 300
  • min(traffic, CAPACITY): the server serves up to its capacity, never more. The rest fail.
  • At 50 and 100 req/s, everyone is served. At 200, half the requests fail. At 400, three out of every four users see an error.
  • Your code isn't wrong. You asked one machine to do more than it can. You need more capacity, and there are only two ways to get it.

Traffic > capacity = errors.

Problem 2: "just buy a bigger server" (vertical scaling)

At the chai stall In your system
A bigger stove: 4 burners → 16 burners More CPU cores: more work at the same time
A bigger fridge: more milk kept close More RAM: more data kept in memory
Upgrade the same stall, no new stalls Vertical scaling ("scale up"): replace your machine with a bigger one

Same stall, just bigger.

Demo 2: scale up

# scale UP: buy a bigger machine (more cores)
PER_CORE = 25          # requests per second per CPU core
TRAFFIC = 400

for cores in [4, 8, 16]:
    capacity = cores * PER_CORE
    failed = max(0, TRAFFIC - capacity)
    print(f"{cores:>2} cores -> capacity {capacity:>3} req/s, failed {failed:>3}")

Output:

 4 cores -> capacity 100 req/s, failed 300
 8 cores -> capacity 200 req/s, failed 200
16 cores -> capacity 400 req/s, failed   0
  • max(0, ...) stops "failed" going negative when capacity is bigger than traffic.
  • 16 cores gives a capacity of 400 and zero failures. It worked!
  • This is a great first move: no code changes at all, you just rent a bigger machine.

A real example: in 2016, Stack Overflow (Nick Craver, Stack Overflow: The Architecture - 2016 Edition) reported 209,420,973 HTTP requests to its load balancer in one day (66,294,789 page loads), served by 11 web servers (9 primary) and 4 SQL Servers. It added that it was down to needing only 1 web server, with the warning: "I'm saying it works. I'm not saying it's a good idea."

The three walls of scaling up

Wall Why it hurts
There's a biggest machine AWS EC2 High Memory U7i instances offer "up to 32TiB of memory and 896 vCPUs". Huge, but a ceiling. Traffic doesn't have one.
Upgrading means downtime The AWS EC2 user guide: "You must stop your instance before you can change its instance type", and "Ensure that you plan for downtime while your instance is stopped." Every upgrade is a small outage.
It's still ONE machine If it dies, everything dies.

The third wall is the scariest. Let's see it.

Problem 3: the big server dies at 2 AM (single point of failure)

However big it is, it's still one machine. Let's see what that means when a disk dies at 2 AM.

Demo 3: one big machine crashes

# one big machine, then it crashes
servers = {"big-server": True}       # True = up

servers["big-server"] = False        # a disk dies at 2 AM

up = [name for name, ok in servers.items() if ok]
print("servers up:", up)
print("handling request on", up[0])

Output:

servers up: []
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    print("handling request on", up[0])
                                 ~~^^^
IndexError: list index out of range
  • The list comprehension [name for name, ok in servers.items() if ok] keeps only the servers that are up. Here, none.
  • up[0] asks for the first item of an empty list: IndexError. There is no server left to take the request.
  • For a real app, every single user is down, not just some. A part whose failure takes down the whole system is called a single point of failure.

Vertical scaling on one page

  • → Replace your machine with a bigger one.
  • + Simple: no code changes.
  • − A ceiling: there's a biggest machine.
  • − Downtime while you upgrade.
  • − Still a single point of failure.

Problem 4: many small servers (horizontal scaling)

On a busy street In your system
Open 5 chai stalls, same recipe in each 5 servers running the same code
Customers spread out: any stall will do Traffic is split: each server takes a share
Busy? Open another. No ceiling, no closing Horizontal scaling ("scale out"): add machines, not a bigger one

Demo 4: scale out, then crash one

# scale OUT: many small machines
PER_SERVER = 100
TRAFFIC = 400
servers = {"s1": True, "s2": True, "s3": True, "s4": True, "s5": True}

servers["s3"] = False                # one machine crashes

up = [name for name, ok in servers.items() if ok]
capacity = len(up) * PER_SERVER
print("servers up:", up)
print(f"capacity {capacity} req/s for {TRAFFIC} req/s of traffic")

# chance that ALL of them are down at once (each up 99%)
print(f"all 5 down: {0.01 ** 5:.10f}")

Output:

servers up: ['s1', 's2', 's4', 's5']
capacity 400 req/s for 400 req/s of traffic
all 5 down: 0.0000000001
  • Same disaster, server s3 crashes, but four servers are still up and together they handle 400 req/s. One crash, and nobody noticed.
  • The last line: if each server is up 99% of the time (down 1% = 0.01) and they fail independently, the chance that all five are down at the same moment is 0.01 ** 5 = 0.0000000001, which is 1 in 10 billion. Remember the nines from last episode? Many ordinary machines can be far more available than one great one.
  • Honest warning: that maths assumes the failures are independent. If all five servers share one power supply, they can die together. That's why big systems spread machines across buildings and cities.

Problem 5: "why am I logged out?" (state in memory)

You order 2 samosas at stall 1. The person there remembers you: 2 samosas, pay later. Then you go to pick it up, but this time you walk into stall 2. Stall 2 says: which order? It has never seen you. What stall 1 remembered stayed in stall 1's head. Servers do exactly the same thing.

A session is what a server remembers about you between requests, like your login and your cart.

Demo 5: sessions in each server's own memory

# each server keeps logged-in users in its OWN memory
class Server:
    def __init__(self, name):
        self.name = name
        self.sessions = {}

    def login(self, user):
        self.sessions[user] = "cart: 2 samosas"
        print(f"{self.name}: {user} logged in")

    def cart(self, user):
        print(f"{self.name}: {user} -> {self.sessions[user]}")

s1, s2 = Server("s1"), Server("s2")
s1.login("riya")      # request 1 lands on s1
s1.cart("riya")       # request 2 lands on s1: fine
s2.cart("riya")       # request 3 lands on s2...

Output:

s1: riya logged in
s1: riya -> cart: 2 samosas
Traceback (most recent call last):
  File "main.py", line 17, in <module>
    s2.cart("riya")       # request 3 lands on s2...
    ~~~~~~~^^^^^^^^
  File "main.py", line 12, in cart
    print(f"{self.name}: {user} -> {self.sessions[user]}")
                                    ~~~~~~~~~~~~~^^^^^^
KeyError: 'riya'
  • Each Server object has its own sessions dictionary. Riya's session was written into s1's dictionary only.
  • When request 3 lands on s2, self.sessions["riya"] doesn't exist: KeyError: 'riya'.
  • In a real app, Riya sees please log in again, and her cart is empty. Add more servers, and it happens more often. These servers are stateful: each one keeps its own memory of users.

Problem 6: one shared store (stateless servers)

On the street In your system
One shared order book: every stall writes in it A shared store: Redis or a database
Any stall can serve you: it just checks the book Stateless servers: keep nothing about you between requests
Add stalls freely: no one's memory is lost Easy scale out: add or remove any server

Redis describes itself as "an open-source, in-memory data structure store" used "as a database, cache, message broker".

Demo 6: stateless servers + a shared store

# fix: servers keep NO session in memory; one shared store holds it
shared_store = {}           # in real life: Redis or a database

class Server:
    def __init__(self, name):
        self.name = name

    def login(self, user):
        shared_store[user] = "cart: 2 samosas"
        print(f"{self.name}: {user} logged in")

    def cart(self, user):
        print(f"{self.name}: {user} -> {shared_store[user]}")

s1, s2, s3 = Server("s1"), Server("s2"), Server("s3")
s1.login("riya")
s2.cart("riya")
s3.cart("riya")

Output:

s1: riya logged in
s2: riya -> cart: 2 samosas
s3: riya -> cart: 2 samosas
  • One change: there's no sessions dictionary inside the server any more. login writes to one shared_store that every server reads.
  • s2 knows her cart, s3 knows her cart. No KeyError, no logout. Any server can serve any request.
  • The big idea: make your servers stateless and push the state into a shared store. Then scaling out is just: start one more copy.

Horizontal scaling on one page

  • → Add more machines running the same code.
  • + No ceiling: just add one more.
  • + One crash ≠ everyone down.
  • − Servers must be stateless.
  • − More moving parts to run.
  • ★ Real systems do both: scale up first while it's cheap and simple, then scale out when you need more room or more safety.

Scale up vs scale out

Vertical · scale up (one bigger machine) Horizontal · scale out (many machines)
Capacity Limited by the biggest machine Add another machine
Upgrade Stop, resize, restart One at a time, no outage
Failure One crash = all down One crash = a small dip
Code No changes needed Servers must be stateless
State In memory is fine In a shared store
In short Simple, but it has a ceiling More work, but it keeps growing

The whole chain

  1. Traffic > capacity → requests fail
  2. Bigger machine → vertical scaling (scale up)
  3. It crashed → single point of failure
  4. More machines → horizontal scaling (scale out)
  5. Logged out → state stuck in one server
  6. Shared store → stateless servers
  7. ? Who sends each request to which server? (In our code we picked s1, s2, s3 by hand.)

Common mistakes (and the real errors)

Mistake Code Error Fix
Every server is down: nothing left to pick up = [] then up[0] IndexError: list index out of range Run more than one server, and check the list isn't empty first
The session lives in another server's memory s2.sessions["riya"] # she logged in on s1 KeyError: 'riya' Keep sessions in a shared store; servers stay stateless
Planning capacity without a real number capacity = len(up) * PER_SERVER # PER_SERVER not set NameError: name 'PER_SERVER' is not defined Measure one server's limit with a load test first

The third one in full:

up = ["s1", "s2"]
capacity = len(up) * PER_SERVER   # PER_SERVER not set
Traceback (most recent call last):
  File "main.py", line 2, in <module>
    capacity = len(up) * PER_SERVER   # PER_SERVER not set
                         ^^^^^^^^^^
NameError: name 'PER_SERVER' is not defined

Misconceptions

  • "Errors under load mean my code has a bug." Often the code is fine; traffic simply went past capacity.
  • "A bigger machine is a safer machine." Bigger ≠ safer. However big it is, one machine is still a single point of failure.
  • "Vertical scaling is always wrong." It's a great first move: simple, no code changes. Teams scale up first, then out.
  • "Five servers means five times the problems." Five independent servers that are each up 99% are all down together far less often than one server is down. (The catch: the failures must really be independent.)
  • "Adding servers is enough." Not if each server keeps sessions in its own memory: users get logged out. Servers must be stateless.

Mini project: the ChaiNow cluster

One program that combines everything: capacity, a crash, stateless servers with a shared store, and what happens when nothing is left. Requests are still picked by hand (request number % number of servers up); next episode a load balancer does this for us.

# ChaiNow cluster: capacity, crashes and stateless servers
PER_SERVER = 100                     # teaching number: req/s per server
shared_store = {}                    # sessions live here, not in a server

servers = {"s1": True, "s2": True, "s3": True, "s4": True, "s5": True}

def up():
    return [name for name, ok in servers.items() if ok]

def report(traffic):
    capacity = len(up()) * PER_SERVER
    failed = max(0, traffic - capacity)
    print(f"  {len(up())} up, capacity {capacity} req/s, traffic {traffic}, failed {failed}")

def handle(request_no, user, action):
    live = up()
    if not live:
        return "503: no server is up"
    server = live[request_no % len(live)]      # picked by hand for now
    if action == "login":
        shared_store[user] = "cart: 2 samosas"
        return f"{server}: {user} logged in"
    return f"{server}: {user} -> {shared_store.get(user, 'please log in')}"

print("Normal day:")
report(400)
print("A disk dies on s3:")
servers["s3"] = False
report(400)
print("Riya's requests:")
for n, action in enumerate(["login", "cart", "cart", "cart"]):
    print(" ", handle(n, "riya", action))
print("Total blackout:")
for name in servers:
    servers[name] = False
print(" ", handle(4, "riya", "cart"))

Output:

Normal day:
  5 up, capacity 500 req/s, traffic 400, failed 0
A disk dies on s3:
  4 up, capacity 400 req/s, traffic 400, failed 0
Riya's requests:
  s1: riya logged in
  s2: riya -> cart: 2 samosas
  s4: riya -> cart: 2 samosas
  s5: riya -> cart: 2 samosas
Total blackout:
  503: no server is up
  • Riya's four requests land on four different servers, and every one knows her cart, because the session lives in shared_store.
  • shared_store.get(user, 'please log in') returns a default instead of raising KeyError.
  • Instead of crashing with IndexError when no server is up, handle checks first and returns a clear error. 503 is the HTTP status code for "Service Unavailable".

Practice

  1. Set TRAFFIC = 1000 with PER_SERVER = 100. How many servers do you need? How many if you also want to survive one crash?
  2. In the scale-out demo (Demo 4), crash two servers. How many requests per second fail?
  3. Add a logout() method to the stateless demo (Demo 6). After Riya logs out on one server, what does another server show?
  4. Each server is up 99%, failing independently. Print the chance that all of them are down for 1, 2, 3 and 5 servers. How many nines does each give?
  5. You run ChaiNow on one AWS EC2 instance and want to change it to a bigger instance type. What must you plan for, and why?
  6. Would you scale ChaiNow up or out first? Why?
Solutions and hints

1. Divide and round up (a part of a server is still a whole server).

import math

PER_SERVER = 100
TRAFFIC = 1000

needed = math.ceil(TRAFFIC / PER_SERVER)
print("servers needed:", needed)
print("with one spare:", needed + 1)
servers needed: 10
with one spare: 11

With exactly 10, one crash drops you to 900 req/s and 100 req/s fail. One spare server lets you lose any one machine and still serve everyone.

2.

# scale OUT: many small machines
PER_SERVER = 100
TRAFFIC = 400
servers = {"s1": True, "s2": True, "s3": True, "s4": True, "s5": True}

servers["s3"] = False                # one machine crashes
servers["s5"] = False                # ...and then another

up = [name for name, ok in servers.items() if ok]
capacity = len(up) * PER_SERVER
print("servers up:", up)
print(f"capacity {capacity} req/s for {TRAFFIC} req/s of traffic")
print("failed:", max(0, TRAFFIC - capacity), "req/s")
servers up: ['s1', 's2', 's4']
capacity 300 req/s for 400 req/s of traffic
failed: 100 req/s

A quarter of the traffic fails, but three quarters is still served. With one big server, a crash means everything fails.

3.

shared_store = {}           # in real life: Redis or a database

class Server:
    def __init__(self, name):
        self.name = name

    def login(self, user):
        shared_store[user] = "cart: 2 samosas"
        print(f"{self.name}: {user} logged in")

    def logout(self, user):
        shared_store.pop(user, None)
        print(f"{self.name}: {user} logged out")

    def cart(self, user):
        session = shared_store.get(user)
        if session is None:
            print(f"{self.name}: {user} -> please log in")
        else:
            print(f"{self.name}: {user} -> {session}")

s1, s2, s3 = Server("s1"), Server("s2"), Server("s3")
s1.login("riya")
s2.cart("riya")
s3.logout("riya")
s1.cart("riya")
s1: riya logged in
s2: riya -> cart: 2 samosas
s3: riya logged out
s1: riya -> please log in

She logged out on s3, and s1 knows it at once, because the session lived in the shared store. pop(user, None) deletes the key without an error if it's already gone.

4.

for n in [1, 2, 3, 5]:
    all_down = 0.01 ** n             # each server up 99%, failing independently
    print(f"{n} server(s): all down {all_down:.10f}  ->  up {(1 - all_down) * 100:.8f}%")
1 server(s): all down 0.0100000000  ->  up 99.00000000%
2 server(s): all down 0.0001000000  ->  up 99.99000000%
3 server(s): all down 0.0000010000  ->  up 99.99990000%
5 server(s): all down 0.0000000001  ->  up 99.99999999%

Each extra server adds two nines (two nines, four, six, ten). Remember this is "at least one server is up"; whether one server has enough capacity for all the traffic is a separate question (see question 1).

5. Downtime. The AWS EC2 user guide says you must stop the instance before changing its type, and to plan for downtime while it is stopped. With only one machine, that means an outage. With several stateless servers, you can upgrade them one at a time.

6. Either answer can be right if you explain it. A common one: scale up first, because it's simple and needs no code changes; move to scaling out when you hit the ceiling, can't accept upgrade downtime, or can't accept a single point of failure. Make the servers stateless before you scale out.

Quick quiz

Plan A: one server, 64 cores. Plan B: four servers, 16 cores each. Same total power. At 2 AM, one machine crashes. Which plan still serves users?

  • A. Plan A
  • B. Plan B
  • C. Both
  • D. Neither
Answer

B, Plan B. Plan B loses a quarter of its capacity but keeps serving. Plan A was one machine, a single point of failure, so everyone is down.

Interview questions

  • What is the difference between vertical and horizontal scaling? Give a real-life analogy.
  • What are the limits of vertical scaling? (A biggest machine, downtime to upgrade, a single point of failure.)
  • What is a single point of failure? How does horizontal scaling remove it, and what assumption does that rely on? (Independent failures.)
  • What does "stateless server" mean? Why must servers be stateless before you scale out?
  • Where do sessions go if not in the server's memory? (A shared store such as Redis or a database.)
  • If each server is up 99% and failures are independent, how likely is it that all three are down at once? (0.01³ = one in a million.)
  • When would you choose to scale up instead of out?
  • How do you work out how many servers you need? (Measure one server's capacity with a load test, divide traffic by it, round up, add spares.)

Glossary

Term Meaning
Server A computer (or program) that answers requests from users' apps
Request One message from an app asking the server to do something, like "show my cart"
req/s (requests per second) How many requests arrive or are handled each second
Capacity The most requests per second a server (or a group of servers) can handle
Traffic How many requests per second users are actually sending
Scaling Growing a system so it can handle more traffic
Vertical scaling (scale up) Replacing your machine with a bigger one: more CPU cores, more RAM
Horizontal scaling (scale out) Adding more machines that run the same code and share the work
CPU core One "worker" inside a processor; more cores can do more work at the same time
vCPU A virtual CPU: a share of a real processor that a cloud provider rents to you
RAM (memory) The fast, temporary storage a computer works from; emptied when the program stops
TiB (tebibyte) A unit of size: 2⁴⁰ bytes, about 1.1 trillion bytes
Instance / instance type A rented cloud machine (on AWS EC2) / its size: how many vCPUs and how much memory
AWS EC2 Amazon's service for renting virtual machines
Downtime / outage Time when a system is not working for users
Single point of failure One part whose failure takes down the whole system
Availability / nines The share of time a system works; 99.99% is "four nines" (episode 4)
Independent failures Failures that don't cause each other, so they rarely happen at the same moment
Session What a server remembers about you between requests: your login, your cart
State Data a program remembers between requests
Stateful server A server that keeps users' state in its own memory
Stateless server A server that keeps nothing about you between requests; any copy can serve you
Shared store One place all servers read and write state, such as Redis or a database
Redis An open-source, in-memory data store, used as a database, cache or message broker
Database A program that stores data safely and lets you search and update it
Load test Sending lots of fake traffic to measure how much one server can really handle
Load balancer The part that decides which server gets each request (next episode)
503 The HTTP status code for "Service Unavailable": no server can handle the request right now
IndexError Python's error for asking for a list position that doesn't exist
KeyError Python's error for reading a dictionary key that isn't there
NameError Python's error for using a variable name that was never set
List comprehension Python's short way to build a list: [x for x in items if test]

Summary

  • Every server has a capacity. Past it, requests fail, even with perfect code.
  • Vertical scaling (scale up) = a bigger machine. Simple, no code changes, but it has a ceiling, needs downtime to upgrade, and is still a single point of failure.
  • Horizontal scaling (scale out) = more machines running the same code. No ceiling, and one crash is only a small dip. Independent machines multiply your nines.
  • Sessions kept in one server's memory break when you scale out (KeyError, user logged out).
  • Fix: stateless servers + a shared store (Redis or a database). Then scaling out is just starting one more copy.
  • In practice: scale up first, then scale out.

Next episode: today we picked servers by hand. Next, we invent the thing that does it for us, millions of times a second: the load balancer. Round robin, least connections, and health checks.