SyntaxPilot
Courses/System Design/Lesson 40
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

System Design · Lesson 40 of 40

Capstone and roadmap

queued

These are the notes for System Design episode 40, the final episode of the course. Every episode of this course ended with a code editor where you ran a program and saw the output. Today you design the system behind that editor. A college holds a national coding contest. At 7 pm the problems open and 50,000 students click Run, almost together. Some of their code is buggy, and some might be trying to break things. Everyone wants a result in seconds, and the contest must be fair. You'll write the requirements, watch one student's infinite loop crash a server that runs code itself (TimeoutError: the code ran for over 2 s), and fix it with a sandbox, limits and verdicts (Time Limit Exceeded, Runtime Error). Then the last student waits 1,000 s, so you add a queue and workers and use Little's law to size the pool: 1,667 busy workers. You'll see that most students first run the same starter template and cache results by hash (102 sandbox runs for 1,000 clicks, 90% saved), stop Ravi's 40 clicks with a token bucket (9 accepted, 31 rejected with 429), and cut the bill with autoscaling (89% cheaper). Then you put every idea of the course in one picture, build a runnable mini code runner, and finish with your roadmap. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). Asha, Ravi, Meera, Neha, Kabir and the other people are fictional. Every number in this episode is our own assumption: the 50,000 clicks in the first minute, 2 s per run, 50 normal workers, Rs 4 per worker hour, the 700/200/100 click mix, the 2-hour peak. The video states no facts about how real companies or real code runners work. Little's law is standard queueing theory.

An online code runner (also called an online judge or an online compiler) is a website where you type a program, click Run, and see its output. Example: the compiler you have watched in every episode of this course.

The problems

Problem What it teaches
The college coding contest Requirements: many languages, verdicts, safe, fast, 50,000 runs in the first minute, never trust what you run
One loop crashes the server Running code on the web server fails; a sandbox with limits and verdicts
50,000 clicks, one runner A job queue plus workers, and Little's law for how many workers
Everyone runs Hello World A result cache keyed by a SHA-256 hash of language, code and input
Ravi clicks Run 40 times A token bucket rate limiter at the gateway (429)
Peak size all day Autoscaling on queue length: 89% cheaper
Put it all together The full design: every box is an episode you already know
The whole course, and what next A course map of all 40 episodes and a 5-step roadmap

Problem 0: the college coding contest (requirements)

Last week Asha solved a contest problem on her laptop. In the real contest, the same site must serve 50,000 students at once, in Python, Java and C++, and any of them can send any code.

A few words first:

  • A coding contest is a timed event where students solve programming problems; each submission is run on hidden test inputs and judged. Example: "print the sum of 1 to n" with n = 9, answer 45.
  • A verdict is the short result a code runner gives for one run, such as OK, Time Limit Exceeded, Runtime Error or Wrong Answer. Example: Ravi's endless loop gets Time Limit Exceeded.
  • Untrusted code is code written by someone you don't know, which you must assume may be buggy or hostile. Example: every student's submission.
  • A stdin (standard input) is the text a program reads while running; its stdout (standard output) is the text it prints. Example: the contest feeds 9 on stdin, and the program prints 45 on stdout.

Functional requirements (what the system must do):

  • Run code in many languages, with input, and show the output and a verdict.
  • Verdicts: OK, Time Limit Exceeded, Runtime Error (and, in the homework, Wrong Answer).

Non-functional requirements (how well it must do it):

  • Safe: one program must never hurt another program or the server.
  • Fast: results in seconds, even at 7 pm.
  • Scale: 50,000 runs in the first minute (our guess).
  • The rule that makes this system special: we run code from strangers. Never trust it.

Problem 1: one loop crashes the server (a sandbox, limits and verdicts)

The obvious design: the web server receives the code and runs it itself. We even add a 2 second timeout. Asha sends a sum, Ravi sends a loop he forgot to stop, Neha sends hello.

  • A subprocess is a separate program started by your program. Python's subprocess.run starts one, waits for it and collects its output. Example: subprocess.run([sys.executable, "-c", code]) starts a new Python that runs the student's code.
  • A timeout is a limit on how long you wait; when it passes, you give up. Example: timeout=2 below.
  • A web server is the program that receives requests from browsers and sends replies. Example: the server behind the Run button.
import signal

def too_slow(signum, frame):
    raise TimeoutError("the code ran for over 2 s")
signal.signal(signal.SIGALRM, too_slow)

def run(name, code):           # the web server runs the code itself
    print(name, end=": ")
    signal.alarm(2)            # a 2 second timer
    exec(code)
    signal.alarm(0)

run("Asha", "print(sum(range(10)))")
run("Ravi", "while True: pass")         # forgot to stop the loop
run("Neha", "print('hello')")           # never reached

Output (file paths shortened to main.py):

Asha: 45
Ravi: Traceback (most recent call last):
  File "main.py", line 14, in <module>
    run("Ravi", "while True: pass")         # forgot to stop the loop
    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "main.py", line 10, in run
    exec(code)
    ~~~~^^^^^^
  File "<string>", line 1, in <module>
  File "main.py", line 4, in too_slow
    raise TimeoutError("the code ran for over 2 s")
TimeoutError: the code ran for over 2 s
  • Asha gets 45 (0 + 1 + ... + 9).
  • Ravi's loop runs for 2 s, the timer fires, and we get TimeoutError: the code ran for over 2 s. The error was never caught, so the whole program crashed, and Neha never gets her answer. A traceback is Python's report of where an error happened, innermost call last.
  • Why? Student code runs inside our own server's program (exec(code) runs text as Python code), and one bad program takes everyone down. Without the timer it would hang for ever. Worse, that code could read our files or use our network.
  • signal.alarm(2) asks the operating system to send our program a signal after 2 s (works on Linux and macOS); too_slow is the function that runs when it arrives. A signal is a message the operating system sends to a running program. An exception such as TimeoutError is Python's way of stopping normal flow when something goes wrong.

Real life: the chemistry lab

In a school lab In a code runner
Experiments happen in a fume hood, not in the classroom A sandbox: a sealed container for each run
"Ten minutes, and only these chemicals" Limits: time, memory, no network
The teacher writes a result for each student: pass, spilled, overtime A verdict: OK, Time Limit Exceeded, Runtime Error
  • A sandbox is a sealed environment where a program runs with strict limits and cannot touch anything outside it. Example: a fresh, empty box per student run, thrown away afterwards.
  • A resource limit caps what a program may use: CPU time, memory, disk, number of processes, network. Example: "at most 2 s of CPU and 256 MB of memory" (our numbers).

The fix: catch what goes wrong, and name it

Every run happens in a sandbox with limits, and we catch what goes wrong: a timeout becomes the verdict Time Limit Exceeded; a crash becomes Runtime Error plus the last line of the error; Meera divides by zero.

import subprocess, sys

def run(code, seconds=2):      # in a sandbox, with limits
    try:
        result = subprocess.run([sys.executable, "-c", code],
                                capture_output=True, text=True, timeout=seconds)
    except subprocess.TimeoutExpired:
        return "Time Limit Exceeded"
    if result.returncode != 0:
        return "Runtime Error: " + result.stderr.strip().splitlines()[-1]
    return "OK: " + result.stdout.strip()

print("Asha:", run("print(sum(range(10)))"))
print("Ravi:", run("while True: pass"))
print("Meera:", run("print(10 / 0)"))
print("Neha:", run("print('hello')"))

Output:

Asha: OK: 45
Ravi: Time Limit Exceeded
Meera: Runtime Error: ZeroDivisionError: division by zero
Neha: OK: hello
  • Asha: OK, 45. Ravi: Time Limit Exceeded. Meera: Runtime Error, ZeroDivisionError: division by zero. Neha: OK, hello. Everyone gets an answer.
  • A returncode is the number a finished program hands back: 0 means success, anything else means it crashed. A ZeroDivisionError is Python's error for dividing by zero; stderr (standard error) is where Python writes error messages, so splitlines()[-1] takes the error's last line.
  • The try / except catches the timeout inside run, so Ravi's loop becomes one line of output instead of a crash. You just invented verdicts.

Real life limits, in general (not any product's design)

In a real system each run usually goes into a container or a small virtual machine, with a memory limit, a CPU limit, no network, and a fresh, empty disk that is thrown away afterwards (general knowledge, described here as general techniques and not as any product's design). That is security from episode 26 and reliability from episode 24.

  • A container is an isolated group of processes that sees its own files and has its own limits, while sharing the machine's operating system. Example: one run inside one container.
  • A virtual machine (VM) is a whole pretend computer running inside a real one; it is a stronger wall than a container but heavier. Example: a small VM per run.

An important warning: a subprocess with a timeout is NOT a security sandbox

The demo above protects us from infinite loops and crashes. It does not protect us from hostile code. The student's program still runs as the same user on the same machine, so it can read our files. Here the "student" reads a file it should never see:

import subprocess, sys, tempfile, os

secret = tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False)
secret.write("the server's private key")       # pretend this is a secret file
secret.close()

def run(code, seconds=2):      # the SANDBOX demo's run(): only a timeout
    result = subprocess.run([sys.executable, "-c", code],
                            capture_output=True, text=True, timeout=seconds)
    return result.stdout.strip()

evil = f"print(open({secret.name!r}).read())"   # student code reads a file it shouldn't
print("student code:", evil.replace(secret.name, "<secret file>"))
print("output:", run(evil))
os.remove(secret.name)

Output:

student code: print(open('<secret file>').read())
output: the server's private key
  • The student's code read the secret file and printed it. A timeout limits time; it does not limit what the program can see or do.
  • That is why the demos in this episode are teaching models. A real runner needs the walls described above: containers or VMs, resource limits, no network, a fresh disk per run (general techniques). Never run strangers' code like this on a real server.

Problem 2: 50,000 clicks, one runner (a queue, workers and Little's law)

Next, speed. 50,000 clicks in the first minute, and say each run takes 2 seconds of CPU (our guess). How long does the last student wait with 100, 1,000 or 2,000 sandboxes?

  • A worker is a program that runs separately, takes a job from a queue and does the slow work. Example: one worker that takes a submission, runs it in a sandbox, and saves the verdict.
  • CPU time is how long a processor actually spends computing for a program. Example: a run that needs 2 s of CPU keeps one core busy for 2 s.
  • Arrival rate is how many jobs arrive per second. Example: 50,000 clicks in 60 s is about 833 a second.
runs = 50_000                  # Run clicks in the contest's first minute
run_time = 2                   # seconds of CPU for one run (our guess)

def last_wait(workers):        # how long the last student waits
    return runs * run_time / workers

for workers in [100, 1_000, 2_000]:
    print(f"{workers:>5,} workers: last result after {last_wait(workers):,.0f} s")

arrival = runs / 60            # runs arriving per second
needed = arrival * run_time    # Little's law: busy workers = rate x time
print(f"to keep up: {arrival:,.0f} runs a second x {run_time} s = {needed:,.0f} busy workers")

Output:

  100 workers: last result after 1,000 s
1,000 workers: last result after 100 s
2,000 workers: last result after 50 s
to keep up: 833 runs a second x 2 s = 1,667 busy workers
  • 50,000 × 2 s = 100,000 seconds of work. With 100 workers that is 100,000 ÷ 100 = 1,000 s, about 17 minutes, in a contest! With 1,000 workers: 100 s. With 2,000: 50 s.
  • The last line is Little's law from queueing theory (the mathematics of waiting lines). In its usual form: the average number of things in a system = the arrival rate × the average time each one spends in it. Here: busy workers = arrival rate × time per job = 833 runs a second × 2 s = 1,667 workers busy at once. Example: a shop where 3 customers arrive each minute and each stays 4 minutes holds about 12 customers at a time. The 833 and the 2 s are our assumptions; the law itself is standard.

Real life: the exam hall counter

At a college exam hall In our system
Students hand in answer sheets at a counter; clerks check them one by one Students' runs are put in a job queue
More clerks mean a shorter line More workers mean a shorter wait
A student does not stand inside the clerk's room The web server never runs code itself

So the web server never runs code. It puts each run in a message queue (episode 16), and a pool of workers, each with sandboxes, takes jobs from it.

  • A message queue is a waiting line of messages: one part of a system adds jobs at the back, another takes them from the front later. Example: ("run #812", code, input).
  • A worker pool is a group of workers taking jobs from the same queue. Example: 1,667 workers busy at 7 pm.
  • The student's page waits for the result, being told when it is ready (the video compares it with the change log push in Drive, episode 38). You just invented a job queue for code.

Problem 3: everyone runs Hello World (a result cache)

Look at what students actually run. Most first click Run on the starter template, unchanged. Say (our assumption) 700 of 1,000 clicks run the same Hello World, 200 run print(2 + 2), and 100 are different.

  • A cache is a fast store of results already computed, so the same question doesn't need the work twice (episodes 7 and 8). Example: the answer to "run print("Hello, World!")" kept for next time.
  • A hash is a fixed-size fingerprint made from data; the same data always gives the same hash. SHA-256 is a standard hash function that makes a 64-character fingerprint (episode 38). Example: hashlib.sha256(b"abc").hexdigest().
  • A cache key is the name you look a cached result up by. Here: the hash of language + code + input.
import hashlib
cache = {}                     # hash(language + code + input) -> result
real_runs = 0

def run(lang, code, stdin):
    global real_runs
    key = hashlib.sha256(f"{lang}\0{code}\0{stdin}".encode()).hexdigest()
    if key not in cache:
        real_runs += 1         # only now do we use a sandbox
        cache[key] = f"output of {code!r}"
    return cache[key]

starter = 'print("Hello, World!")'       # everyone runs the template first
clicks = [starter] * 700 + ["print(2 + 2)"] * 200 + [f"print({i})" for i in range(100)]
for code in clicks:
    run("python", code, "")
print(f"Run clicks: {len(clicks):,}")
print(f"sandbox runs: {real_runs:,}")
print(f"saved: {1 - real_runs / len(clicks):.0%}")

Output:

Run clicks: 1,000
sandbox runs: 102
saved: 90%
  • 1,000 clicks, only 102 sandbox runs: 1 for the template, 1 for print(2 + 2), and 100 different programs. 1 - 102/1,000 = 0.898, shown as 90% saved.
  • The key includes the language, the code and the input, because changing any of them can change the output. The \0 (a zero byte) between them keeps "ab" + "c" from looking the same as "a" + "bc".
  • In this demo the "run" just builds a string; in a real runner cache[key] = ... would hold the verdict and output a worker produced.
  • The catch: caching is only safe for programs that give the same output every time (they are deterministic). Code that prints the time or a random number must skip the cache (see Common mistakes). You just invented a result cache.

Problem 4: Ravi clicks Run 40 times (a token bucket)

Ravi is nervous, so he clicks Run four times a second for ten seconds: 40 runs. Multiply that by a few thousand nervous students and the queue fills with junk. We use a token bucket (episode 19): a bucket holds 5 tokens, each run costs one, and one token comes back every 2 seconds.

  • A rate limiter caps how often one user may call a service. Example: at most a few runs per second per student.
  • A token bucket is a rate limiter that lets short bursts through but holds a steady average: you may spend a token only if the bucket has one, and tokens refill at a fixed pace. Example: 5 runs at once, then one more every 2 s.
  • HTTP 429 (Too Many Requests) is the web's standard reply that says "slow down". Example: Ravi's 10th click gets 429.
import time
class TokenBucket:             # episode 19: 5 runs, refilled 1 every 2 s
    def __init__(self, size=5, refill_every=2):
        self.size, self.tokens, self.every = size, size, refill_every
        self.last = time.time()
    def allow(self, now):
        self.tokens = min(self.size, self.tokens + (now - self.last) / self.every)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

bucket = TokenBucket()
start = bucket.last
results = [bucket.allow(start + t * 0.25) for t in range(40)]   # 4 clicks a second, 10 s
print("Ravi clicks Run 40 times in 10 s")
print("accepted:", results.count(True), " rejected (429):", results.count(False))

Output:

Ravi clicks Run 40 times in 10 s
accepted: 9  rejected (429): 31
  • Accepted: 9. Rejected with 429: 31. The bucket starts with 5 tokens (5 runs at once), and over the 9.75 s of clicks about 4.9 more tokens refill (one per 2 s), so 5 + 4 whole tokens spent = 9.
  • The clicks are given fake times (start + t * 0.25), so the result is the same every run; a real limiter would pass the current time.
  • Ravi still gets his results, and his clicks can't starve everyone else. The limit lives in the API gateway (episode 20), before anything reaches the queue. You just invented fairness.
  • An API gateway is the front door of a system: it checks who you are and how often you call, then passes the request on. Example: the box in front of the run service.

Problem 5: peak size all day (autoscaling)

Last problem: money. We need 1,667 workers at 7 pm, but maybe 50 on a normal evening (our guess). Say a worker costs Rs 4 an hour (a made-up number). Compare keeping the peak size all day with autoscaling for a 2 hour peak.

  • Horizontal scaling means adding more machines instead of making one bigger (episode 5). Example: going from 50 workers to 1,667.
  • Autoscaling means adding and removing machines automatically as load changes. Example: more workers when the queue grows, fewer when it empties.
  • Queue length (backlog) is how many jobs are waiting. Example: 20,000 runs waiting is a signal to add workers.
peak_workers = 1_667           # from the contest minute (our estimate)
normal_workers = 50            # a normal evening (our guess)
cost_per_worker_hour = 4       # rupees, our made-up number

always_peak = peak_workers * 24 * cost_per_worker_hour
autoscale = (normal_workers * 22 + peak_workers * 2) * cost_per_worker_hour
print(f"peak size all day: Rs {always_peak:,} a day")
print(f"autoscaling (2 peak hours): Rs {autoscale:,} a day")
print(f"saved: {1 - autoscale / always_peak:.0%}")

Output:

peak size all day: Rs 160,032 a day
autoscaling (2 peak hours): Rs 17,736 a day
saved: 89%
  • Peak size all day: 1,667 × 24 × Rs 4 = Rs 160,032 a day (about 1.6 lakh). Autoscaling: (50 × 22 h + 1,667 × 2 h) × Rs 4 = Rs 17,736. 89% cheaper. A lakh is 100,000 (an Indian number word).
  • Autoscaling watches the queue length and adds workers when it grows. And because the contest start time is known, we start workers about ten minutes before 7 pm, not after the queue has exploded (new workers take time to start).
  • You just invented autoscaling. All the money numbers are made up.

The full design

  students      API gateway          run service            queue           workers          sandboxes
  ────────      ───────────          ───────────            ─────           ───────          ─────────
  browser ───▶ login, rate limit ──▶ check result cache ──▶ per language ──▶ autoscaled ───▶ time, memory,
  (Run)         (token bucket,        hit  → answer now      (job id)        (take jobs)     no network,
     ▲           429)                 miss → job id                              │           fresh disk
     │                                                                          ▼
     └──────── result pushed over a WebSocket ◀───────────────────────── result store (cache + DB)
                          logs, metrics, traces (episode 25) watch every box
  1. Students' browsers talk to an API gateway that checks login and the rate limit.
  2. The run service checks the result cache. On a hit it answers at once. On a miss it gives the run a job id and puts it in a queue per language.
  3. Autoscaled workers take jobs, run each in a fresh sandbox with time and memory limits and no network, and save the result to the result store and the cache.
  4. Results are pushed back over a WebSocket to the student's page. Logs, metrics and traces (episode 25) show where time is going on contest night.
  5. Look again: a gateway, rate limiting, caching, queues, workers, scaling, security, observability. Every box is an episode you already know. That is the whole point of this course.
  • A WebSocket is a long-lived connection between browser and server over which the server can push messages whenever it likes. Example: "your result is ready" arrives without the page asking.
  • A job id is a unique name for one run, so the page and the worker can find each other's result. Example: run-812.
  • A result store is a database that keeps finished verdicts and outputs. Example: where "Meera: Runtime Error" is saved.
  • Observability means being able to see what a system is doing from the outside, using logs, metrics and traces (episode 25).

The whole chain

  1. TimeoutError: the code ran for over 2 s took the server down → a sandbox with limits and verdicts.
  2. The last result after 1,000 s → a queue and workers; Little's law says 1,667 busy workers.
  3. 700 identical runs → a result cache keyed by hash: 90% saved.
  4. 40 clicks from Ravi → a token bucket: 9 accepted, 31 rejected (429).
  5. Peak size all day → autoscaling: 89% cheaper.

Common mistakes (and the real outputs)

Mistake What you see Why Fix
Run student code on the web server TimeoutError: the code ran for over 2 s One bad program takes everyone down Sandbox + limits + verdicts, on workers
Too few workers for the rush 100 workers: last result after 1,000 s Arrival rate × time > workers Little's law, a queue, autoscaling
Cache every program's output time shown by the second run minus the first: 0 (2 s passed) Not every program gives the same output each time Cache only programs with the same output every time
Think a timeout makes a sandbox output: the server's private key A timeout limits time, not what the code can read Containers or VMs, no network, fresh disk
No Wrong Answer verdict Kabir: OK: 36 (the right answer is 45) The runner only checks that the program did not crash Compare the output with the expected output

The "cache everything" row is a cache with the key made from the code only, applied to a program that prints the current time:

import hashlib, subprocess, sys, time
cache = {}

def run(code):
    key = hashlib.sha256(code.encode()).hexdigest()
    if key not in cache:                      # cache EVERYTHING: the mistake
        cache[key] = subprocess.run([sys.executable, "-c", code],
                                    capture_output=True, text=True).stdout.strip()
    return cache[key]

code = "import time; print(int(time.time()))"      # prints the current time
first = int(run(code))
time.sleep(2)
second = int(run(code))
print("seconds between the two clicks: 2")
print("time shown by the second run minus the first:", second - first)

Output:

seconds between the two clicks: 2
time shown by the second run minus the first: 0

The second click came 2 seconds later, but the student sees the old time: the cache returned the first answer.

The "no Wrong Answer verdict" row is the verdict demo with an input and a buggy answer (range(n) stops one short, so it adds 0 to 8 and gets 36, not 45):

import subprocess, sys

def run(code, stdin="", seconds=2):    # the SANDBOX demo's run(), plus input
    try:
        result = subprocess.run([sys.executable, "-c", code], input=stdin,
                                capture_output=True, text=True, timeout=seconds)
    except subprocess.TimeoutExpired:
        return "Time Limit Exceeded"
    if result.returncode != 0:
        return "Runtime Error: " + result.stderr.strip().splitlines()[-1]
    return "OK: " + result.stdout.strip()

# the task: print 1 + 2 + ... + n.  For n = 9 the right answer is 45.
print("Kabir:", run("n = int(input()); print(sum(range(n)))", "9"))

Output:

Kabir: OK: 36

Mini project: a tiny online code runner

A job queue, three worker threads, a subprocess sandbox with a 1 s timeout, four verdicts (OK, Wrong Answer, Time Limit Exceeded, Runtime Error), a result cache keyed by a SHA-256 hash of language, code and input, and a per-user token bucket (3 runs, one more every 2 s). Time for the token bucket is passed in (simulated), so the output is the same on every run. The task is "read n, print 1 + 2 + ... + n"; the test input is 9 and the expected output is 45.

A thread is a path of execution inside a program; several threads can run at the same time. Example: three worker threads taking jobs from one queue. A lock lets only one thread at a time change shared data. Example: with lock: around the cache.

Remember the warning: this runner's sandbox is a subprocess with a timeout, a teaching model, not a security boundary.

# Mini project: a tiny online code runner. A queue, three worker threads, a
# subprocess "sandbox" with a timeout, verdicts (OK, Wrong Answer, Time Limit
# Exceeded, Runtime Error), a result cache and a per-user token bucket.
# NOTE: a subprocess with a timeout is NOT a security sandbox (see the notes).
import hashlib, queue, subprocess, sys, threading

TIMEOUT = 1                                   # seconds a run may use (ours)
WORKERS = 3                                   # worker threads (ours)
BUCKET, REFILL = 3, 2                         # 3 runs, +1 every 2 s (ours)

class TokenBucket:                            # episode 19, one bucket per user
    def __init__(self):
        self.tokens, self.last = BUCKET, 0.0
    def allow(self, now):
        self.tokens = min(BUCKET, self.tokens + (now - self.last) / REFILL)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

def sandbox(code, stdin, expected):           # run one program, return a verdict
    try:
        r = subprocess.run([sys.executable, "-c", code], input=stdin,
                           capture_output=True, text=True, timeout=TIMEOUT)
    except subprocess.TimeoutExpired:
        return "Time Limit Exceeded"
    if r.returncode != 0:
        return "Runtime Error: " + r.stderr.strip().splitlines()[-1]
    if r.stdout.strip() != expected:
        return "Wrong Answer"
    return "OK"

jobs = queue.Queue()                          # the job queue
results, cache, buckets = {}, {}, {}
sandbox_runs = 0
lock = threading.Lock()

def worker():                                 # takes jobs until it gets None
    global sandbox_runs
    while True:
        job = jobs.get()
        if job is None:
            return
        sub_id, key, code, stdin, expected = job
        verdict = sandbox(code, stdin, expected)
        with lock:
            sandbox_runs += 1
            cache[key] = verdict
            results[sub_id] = verdict
        jobs.task_done()

def submit(sub_id, user, now, code, stdin="9", expected="45"):   # the API
    if not buckets.setdefault(user, TokenBucket()).allow(now):
        results[sub_id] = "429 Too Many Requests"
        return
    key = hashlib.sha256(f"python\0{code}\0{stdin}".encode()).hexdigest()
    with lock:
        hit = cache.get(key)
    if hit:                                   # same language+code+input: no sandbox
        results[sub_id] = hit + " (cached)"
        return
    jobs.put((sub_id, key, code, stdin, expected))

threads = [threading.Thread(target=worker) for _ in range(WORKERS)]
for t in threads:
    t.start()

GOOD = "n = int(input()); print(sum(range(1, n + 1)))"       # sum 1..n
WAVE1 = [("Asha", GOOD), ("Ravi", "while True: pass"), ("Meera", "print(10 / 0)"),
         ("Kabir", "n = int(input()); print(sum(range(n)))"), ("Neha", GOOD)]
order = []
for i, (user, code) in enumerate(WAVE1, 1):   # wave 1: five students click Run
    order.append((f"#{i}", user))
    submit(f"#{i}", user, now=0.0, code=code)
jobs.join()                                   # wait until wave 1 is done
for i in range(6, 12):                        # wave 2: Ravi hammers Run, 0.5 s apart
    order.append((f"#{i}", "Ravi"))
    submit(f"#{i}", "Ravi", now=1.0 + (i - 6) * 0.5, code=GOOD)
order.append(("#12", "Asha"))                 # and Asha reruns her good code
submit("#12", "Asha", now=4.0, code=GOOD)
jobs.join()
for _ in threads:
    jobs.put(None)
for t in threads:
    t.join()

for sub_id, user in order:
    print(f"{sub_id:<4}{user:<6}{results[sub_id]}")
print(f"clicks: {len(order)}, sandbox runs: {sandbox_runs}, "
      f"cached: {sum('(cached)' in v for v in results.values())}, "
      f"rejected: {sum(v.startswith('429') for v in results.values())}")

Output:

#1  Asha  OK
#2  Ravi  Time Limit Exceeded
#3  Meera Runtime Error: ZeroDivisionError: division by zero
#4  Kabir Wrong Answer
#5  Neha  OK
#6  Ravi  OK (cached)
#7  Ravi  OK (cached)
#8  Ravi  OK (cached)
#9  Ravi  429 Too Many Requests
#10 Ravi  429 Too Many Requests
#11 Ravi  429 Too Many Requests
#12 Asha  OK (cached)
clicks: 12, sandbox runs: 5, cached: 4, rejected: 3
  • Four verdicts in wave 1: Asha OK, Ravi Time Limit Exceeded (killed after 1 s), Meera Runtime Error: ZeroDivisionError: division by zero, Kabir Wrong Answer (his code printed 36, not 45). Three workers ran the five jobs side by side, so the whole program took about one second, not five.
  • Cache: Ravi's good code (#6 to #8) and Asha's rerun (#12) came from the cache, with no sandbox. Neha's #5 was not cached: Asha's and Neha's identical code arrived in the same wave, before the first result was saved, so both ran (5 sandbox runs for 12 clicks). A real system would also remember jobs that are in flight so the second click waits for the first (see "Try extending it").
  • Token bucket: Ravi's bucket held 3 tokens. His loop (#2) used one, and #6 to #8 used the rest as tokens trickled back; #9 to #11 got 429. The limit is checked before the cache, so even a cached rerun costs a token.
  • Queue + workers + results by id: the API only puts jobs in the queue; the workers write results[sub_id], which the student's page would read.

Try extending it: remember in-flight jobs so duplicate clicks in the same wave share one run; skip the cache for code that imports time or random; add a Memory Limit Exceeded verdict with the resource module (on Linux); run a separate queue per language; and make the workers add themselves when jobs.qsize() grows (autoscaling).

The whole course, in one map

Look how far you've come. Here are all 40 episodes, grouped as in the video.

Group Episodes Titles
How the web works 1-4 1 What happens when you open a website; 2 Client–server and HTTP; 3 DNS deep dive; 4 Latency vs throughput
Scaling and caching 5-8 5 Scaling: vertical vs horizontal; 6 Load balancers; 7 Caching; 8 Cache strategies
Data at scale 9-15 9 Databases: SQL vs NoSQL; 10 Indexing; 11 Replication; 12 Sharding; 13 Consistent hashing; 14 CAP and PACELC; 15 Consistency models
Services talking 16-20 16 Message queues and pub/sub; 17 Monolith vs microservices; 18 API design: REST, GraphQL, gRPC; 19 Rate limiting; 20 Proxies and API gateways
Building blocks 21-26 21 Storage and CDNs; 22 Search: the inverted index; 23 Unique IDs at scale; 24 Reliability patterns: retries, idempotency, circuit breakers; 25 Observability: logs, metrics, traces; 26 Security basics: auth, OAuth, JWT, encryption
Real designs + interviews 27-39 27 Design a URL shortener; 28 Design a rate limiter service; 29 Design WhatsApp chat; 30 Design the Instagram feed; 31 Design Hotstar live streaming for IPL; 32 Design Uber/Ola ride matching; 33 Design Swiggy/Zomato delivery; 34 Design UPI payments; 35 Design IRCTC Tatkal booking; 36 Design search autocomplete; 37 Design a notification system; 38 Design a file storage service (Drive); 39 The system design interview framework
Capstone 40 40 Capstone and roadmap (this episode)

And where this episode used them:

Box in the code runner Episode(s)
Queue between the run service and workers 16
Result cache, cache by hash 7, 8, 38
Rate limit (token bucket) 19
API gateway 20
Horizontal scaling and autoscaling 5
Retries and reliability 24
Logs, metrics, traces 25
Security (sandbox, no network) 26

Your roadmap

  1. Redo three designs from memory, out loud, in 45 minutes each. Pick three designs from episodes 27 to 38 and explain them to an empty room, or to a friend.
  2. Do mock interviews with a friend, and swap roles. Being the interviewer teaches you a lot.
  3. Build one for real. A small URL shortener with a cache and a database. Then load test it (send it many requests at once) and watch what breaks.
  4. Read company engineering blogs, where teams explain how they really scaled. Treat each as one team's story, not a universal rule.
  5. Go deeper into databases, distributed systems and networks.

And the habit that makes a great designer: for every app you use, ask "what breaks at 10x the users?"

Practice

  1. Homework: in the SANDBOX demo, add a Wrong Answer verdict by comparing the output with the expected one.
  2. Homework: in the WORKERS demo, runs take 5 s. How many workers? What does the SCALE bill become?
  3. In the CACHE demo, 102 of 1,000 clicks reach a sandbox. How many busy workers do we need now at the contest minute?
  4. In the LIMIT demo, the bucket holds 10 runs instead of 5. How many of Ravi's 40 clicks are accepted?
  5. Little's law: runs arrive at 100 a second and take 0.5 s; or 2,000 a second and 4 s. How many busy workers?
  6. Why must code that prints the time or a random number skip the cache?
  7. Why is a subprocess with a timeout not a security sandbox? What does a real runner add (in general)?
  8. Why start the workers ten minutes before 7 pm, instead of waiting for the queue to grow?
Solution

1. Give run an input and an expected output, and add a third check after the crash check:

import subprocess, sys

def run(code, stdin="", expected=None, seconds=2):   # in a sandbox, with limits
    try:
        result = subprocess.run([sys.executable, "-c", code], input=stdin,
                                capture_output=True, text=True, timeout=seconds)
    except subprocess.TimeoutExpired:
        return "Time Limit Exceeded"
    if result.returncode != 0:
        return "Runtime Error: " + result.stderr.strip().splitlines()[-1]
    if expected is not None and result.stdout.strip() != expected:
        return "Wrong Answer"
    return "OK: " + result.stdout.strip()

# the task: read n, print the sum 1 + 2 + ... + n.  For n = 9 the answer is 45.
print("Asha:", run("n = int(input()); print(sum(range(1, n + 1)))", "9", "45"))
print("Ravi:", run("while True: pass", "9", "45"))
print("Meera:", run("print(10 / 0)", "9", "45"))
print("Kabir:", run("n = int(input()); print(sum(range(n)))", "9", "45"))   # off by one: 36

Output:

Asha: OK: 45
Ravi: Time Limit Exceeded
Meera: Runtime Error: ZeroDivisionError: division by zero
Kabir: Wrong Answer

The order matters: a run that timed out or crashed has no trustworthy output, so check those first; only a clean run is compared with the expected output. Real judges also usually ignore trailing spaces and blank lines (we use .strip()).

2. Change run_time = 2 to run_time = 5. Output:

  100 workers: last result after 2,500 s
1,000 workers: last result after 250 s
2,000 workers: last result after 125 s
to keep up: 833 runs a second x 5 s = 4,167 busy workers

833 × 5 = 4,167 busy workers (2.5 times more than 1,667, because each run is 2.5 times longer). In the SCALE demo, set peak_workers = 4_167. Output:

peak size all day: Rs 400,032 a day
autoscaling (2 peak hours): Rs 37,736 a day
saved: 91%

Autoscaling saves even more (91%), because the peak is further above the normal 50 workers.

3. Only the cache misses need a worker:

arrival = 50_000 / 60          # clicks a second
run_time = 2                   # seconds per run
hit_rate = 1 - 102 / 1000      # from the cache demo: 102 real runs in 1,000 clicks
real_runs_per_second = arrival * (1 - hit_rate)
print(f"workers without a cache: {arrival * run_time:,.0f}")
print(f"workers with the cache:  {real_runs_per_second * run_time:,.0f}")

Output:

workers without a cache: 1,667
workers with the cache:  170

About 170 workers instead of 1,667, if the same 90% hit rate holds at the contest minute (our assumption: the click mix is made up).

4. Change size=5 to size=10. Output:

Ravi clicks Run 40 times in 10 s
accepted: 14  rejected (429): 26

The bigger bucket lets 14 through (10 at once + 4 refilled). A bigger bucket allows a bigger burst but the long-run rate (one every 2 s) is unchanged.

5. Busy workers = arrival × time:

def busy(arrival, seconds): return arrival * seconds
print(busy(500, 3), busy(100, 0.5), busy(2000, 4))

Output:

1500 50.0 8000

500 × 3 = 1,500 (the quiz), 100 × 0.5 = 50, 2,000 × 4 = 8,000.

6. The cache returns the old output for the same code. A program that prints time.time() or a random number should give a new answer each time, but a cached run returns the first one (see the Common mistakes demo: the second run showed the old time, difference 0). Cache only deterministic programs, or skip the cache for code that uses time, randomness or the network.

7. The timeout only limits how long a program runs. The program still runs as the same user on the same machine, so (as the demo showed) it can read files, and it could also use the network or start many processes. A real runner adds, in general: a container or small VM per run, CPU and memory limits, no network, a fresh empty disk thrown away afterwards, and a limit on the number of processes. These are general techniques; the demos here are teaching models.

8. New workers take time to start (machines boot, programs load). If you wait for the queue to grow, students are already waiting before help arrives. The contest time is known in advance, so you can start workers early and keep scaling on queue length for surprises.

Quick quiz

arrival = 500   # runs a second
run_time = 3    # seconds each
workers = ???

Runs arrive at 500 a second and take 3 s each. How many workers are busy at once?

  • A. 167
  • B. 503
  • C. 1,500
  • D. 15,000
Answer

C. Little's law: busy workers = arrival rate × time per job = 500 × 3 = 1,500.

Interview questions: "Design an online code runner / online judge"

1. Clarify requirements. Which languages? Is it practice (a few users) or a live contest (a huge burst)? Do we only run code, or also judge it against hidden tests with a verdict? Input and output sizes? Time and memory limits per run? How fast must results come back? Then state: run untrusted code safely, give a verdict, fair under a burst, results in seconds.

2. Estimate (our assumptions, said out loud). "50,000 runs in the first minute, 2 s each. That is about 833 runs a second. By Little's law, busy workers = 833 × 2 ≈ 1,667. If 70% of clicks are the starter template, a cache cuts that sharply. The peak lasts maybe two hours, so I would autoscale instead of paying for the peak all day."

3. Components.

  • API gateway: login, per-user rate limit (token bucket, 429).
  • Run service: checks the result cache (key = hash of language, code, input), makes a job id, enqueues.
  • Queues: per language, so one slow language doesn't block another.
  • Workers, autoscaled: take jobs, run each in a fresh sandbox, compare output with the expected output, produce a verdict.
  • Sandbox: container or small VM with CPU, memory and process limits, no network, fresh disk (general technique).
  • Result store and cache; WebSocket push of results; logs, metrics, traces.

4. The hard parts. Running untrusted code safely (isolation, not just a timeout); a huge burst at a known time (queue + workers + pre-scaling); fairness (rate limit per user); not wasting work (cache by hash; skip it for non-deterministic code); and an honest verdict (time limit, runtime error, wrong answer).

5. Scale and reliability (general knowledge). Stateless API servers behind a load balancer; replicate the queue so a crashed machine loses no jobs; a worker that dies mid-run lets the job be handed out again (idempotent runs, episode 24); alert when the queue length or the worst wait grows; keep a dead letter queue for runs that keep crashing the sandbox itself.

Common follow-ups:

  • What if a student's code forks many processes or fills the disk? (A process limit, a disk quota, and a fresh disk per run.)
  • What if two identical submissions arrive at the same moment? (Remember in-flight jobs so the second waits for the first.)
  • How do you keep the contest fair when some runs are slower? (Same limits for everyone; judge by CPU time, not queue waiting time.)
  • How do you test the design before the contest? (A load test that sends the expected burst, and watching the queue.)
  • A worker crashes after finishing but before saving the result? (The job is handed out again; the result is the same, so saving it twice is harmless.)

Glossary

Term Meaning
Online code runner / judge A website where you type a program, click Run, and see its output and a verdict
Coding contest A timed event where students solve programming problems and submissions are judged
Verdict The short result of a run: OK, Time Limit Exceeded, Runtime Error, Wrong Answer
Time Limit Exceeded The program ran longer than allowed and was stopped, e.g. an infinite loop
Runtime Error The program crashed while running, e.g. ZeroDivisionError: division by zero
Wrong Answer The program finished cleanly but its output differs from the expected output
Untrusted code Code from someone you don't know, assumed possibly buggy or hostile
stdin / stdout / stderr A program's input text / its printed output / where error messages are written
Returncode The number a finished program returns; 0 means success
Subprocess A separate program started by your program, e.g. subprocess.run
Timeout A limit on how long you wait before giving up
TimeoutError Python's error for a timeout; in the first demo, raised by our own 2 s timer
subprocess.TimeoutExpired Python's error when a subprocess runs past its timeout (the sandbox demo catches it)
exec / signal Python's exec runs text as code / a message the operating system sends to a running program
Traceback Python's report of where an error happened, innermost call last
Web server The program that receives browser requests and sends replies
Sandbox A sealed environment where a program runs with strict limits and cannot touch the outside
Resource limit A cap on CPU time, memory, disk, processes or network for a program
Container An isolated group of processes with its own files and limits, sharing the machine's operating system
Virtual machine (VM) A whole pretend computer running inside a real one; a stronger wall, but heavier
Message queue A waiting line of jobs: added at the back, taken from the front later
Worker / worker pool A separate program that takes jobs from the queue and does the slow work / a group of them
CPU time How long a processor actually computes for a program
Arrival rate How many jobs arrive per second, e.g. 833 runs a second
Little's law In a stable system, average number inside = arrival rate × average time inside; here busy workers = runs per second × seconds per run
Queueing theory The mathematics of waiting lines
Cache A fast store of results already computed, so the work isn't repeated
Cache key The name a cached result is looked up by, here the hash of language + code + input
Hash / SHA-256 A fixed-size fingerprint of data / a standard hash function that gives a 64-character fingerprint
Deterministic Gives the same output every time for the same input; only such programs are safe to cache
Rate limiter A cap on how often one user may call a service
Token bucket A rate limiter with a bucket of tokens: each request spends one, tokens refill at a fixed pace, short bursts are allowed
HTTP 429 "Too Many Requests", the web's reply that says "slow down"
API gateway The front door of a system: checks login and rate limits, then passes requests on
Horizontal scaling Adding more machines instead of making one bigger
Autoscaling Adding and removing machines automatically as load changes
Queue length / backlog How many jobs are waiting; the signal for autoscaling
Lakh 100,000 (Indian number word); 1.6 lakh = 160,000
Job id A unique name for one run, so the page and the worker can find each other's result
Result store A database that keeps finished verdicts and outputs
WebSocket A long-lived browser-server connection over which the server can push messages
Observability Seeing what a system is doing from outside, using logs, metrics and traces
Thread / lock A path of execution inside a program / a rule that lets one thread at a time change shared data
Load test Sending a system many requests at once to see where it breaks
Peak The busiest period of load, here the contest's first minutes
Roadmap A plan of what to learn or do next

Summary

  • An online code runner runs untrusted code, so never run it on the web server: one while True: pass gave TimeoutError and crashed everything. Run each program in a sandbox with limits and report a verdict: OK, Time Limit Exceeded, Runtime Error, Wrong Answer. A subprocess with a timeout is not a security sandbox (it read a secret file); real runners use containers or VMs, resource limits, no network and a fresh disk.
  • Put each run in a queue and let workers take jobs. Little's law: busy workers = arrival rate × time per job = 833 × 2 s = 1,667 (100 workers: the last student waits 1,000 s). All numbers are our assumptions.
  • A result cache keyed by the SHA-256 hash of language, code and input: 102 sandbox runs for 1,000 clicks, 90% saved. Never cache programs whose output changes (time, random).
  • A token bucket at the API gateway: Ravi's 40 clicks, 9 accepted, 31 rejected with 429.
  • Autoscaling on queue length, started before the known 7 pm peak: Rs 17,736 vs Rs 160,032 a day, 89% cheaper (made-up prices).
  • Every box in the full design is an episode of this course. Your roadmap: redo three designs from memory, mock interviews, build one for real, read engineering blogs, go deeper, and always ask "what breaks at 10x?".

This is the final episode. Forty episodes, from opening a website to designing Drive and a code runner. Keep designing, keep building.

Sources: none needed. This episode states no facts about real companies or real code runners. All numbers are our own assumptions: the 50,000 clicks in the first minute, 2 s (and 5 s) per run, 50 normal workers, Rs 4 per worker hour, the 700/200/100 click mix, the 2-hour peak, the token bucket of 5 (and 10), the mini project's limits, and all names. Little's law, queueing theory, sandboxes, containers, virtual machines, hashes, token buckets and autoscaling are general textbook knowledge.