SyntaxPilot
Courses/System Design/Lesson 38
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

System Design · Lesson 38 of 40

Design a file storage service (Drive)

queued

These are the notes for System Design episode 38. Priya films her cousin's wedding on her phone: one video, 2 GB. She wants it safe in the cloud (uploaded from a train), found on her laptop at home, and shared with her mom and her brother Rahul. Today you design the file storage service behind that: Google Drive, Dropbox. You'll write the requirements, watch a one-go upload thrown away three times by train tunnels (ConnectionResetError: upload failed, start again) and fix it with chunks and a resumable upload, notice that fixing one word re-sends the whole file and fix it with blocks and SHA-256 hashes (2 of 9 blocks), stop three relatives from filling the disks with the same video by deduplication (1,000,000 bytes stored, not 3,000,000), split the data into a metadata database and a block store, stop a last write wins save from silently losing "Agra" with versions and conflicted copies, replace 172 billion polls a day (only 0.12% useful) with a change log, a cursor and a push, and size it all: 250 PB of user data becomes 525 PB of disks. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). Priya, Rahul and the other people are fictional. The 8% network drop, the 100 MB chunks, the 4-byte demo blocks, 50 million users with 5 GB each, 30% duplicates, 3 copies, 10 million devices polling every 5 seconds and 20 changes a day are our own assumptions; the real-world facts come only from the two sources at the end.

A file storage service keeps your files on a company's servers (the "cloud"), so you can open them from any of your devices and share them with other people. Example: a photo saved on Priya's phone appears on her laptop a second later.

The problems

Problem What it teaches
Priya's wedding video Requirements: upload, sync, share; never lose a file or an edit; store as little as possible
A tunnel kills the upload All-or-nothing uploads waste everything; chunks and a resumable upload
One word changed, whole file sent Blocks, hashes (SHA-256), uploading only the changed blocks
The same video, three times Deduplication: a block store keeps each block once; a file is a list of hashes
Where does the file list live? Metadata database vs block store; blocks first, then commit
Two devices, one file Last write wins loses edits; versions and conflicted copies
Asking every 5 seconds Polling is wasteful; a change log, a cursor and a push
The scale 50 million users, 5 GB each: 250 PB, 175 PB after dedup, 525 PB of disks
Put it all together Client, API servers, metadata DB, block store, change log, notifications, CDN

Problem 0: Priya's wedding video (requirements)

Priya's one video has three jobs, and each one hides a problem:

Job What Priya wants The hidden problem
Upload The 2 GB video safe in the cloud, sent from a train Networks drop
Sync The video on her laptop at home, without doing anything Which device has the newest copy?
Share Mom and Rahul can open it Many copies of the same bytes

A few words first:

  • A server is a computer, owned by the service, that waits for requests and answers them. A client is the app on your phone or laptop that sends the requests. Example: the Drive app is a client; Google's computers are the servers.
  • Upload means sending a file from your device to the server; download is the other way round. Sync (synchronise) means keeping the same files on all your devices up to date. Share means letting other people open your file.
  • A byte is the smallest chunk of data we count (one letter of plain English text is one byte). KB, MB, GB, TB, PB are kilobyte, megabyte, gigabyte, terabyte and petabyte: each one is 1,000 times the one before it in this course (a rough rule; computers sometimes use 1,024). Example: 2 GB = 2,000 MB.
  • Offline means a device that is not connected to the internet right now. Example: Priya's laptop in a train tunnel.

Functional requirements (what the system must do):

  • Upload and download files, even huge ones. For a sense of scale, Google's admin help says: "You can upload and synchronize files up to 5 TB in size."
  • Sync across phone, laptop and web.
  • Share files and folders, and keep old versions.

Non-functional requirements (how well it must do it):

  • Durable: never lose a file, and never lose an edit. Durability means data survives crashes and broken disks.
  • Works on bad networks, and sends as little as possible.
  • The money one: storage is the biggest cost, so store as little as possible. (Our reasoning, and the reason the rest of the design exists.)

Real services also put limits on use. Google's admin help says: "Each user can upload and copy 750 GB to Drive within 24 hours." (Practice 8 turns that into a number of videos.)

Problem 1: a tunnel kills the upload (chunks and a resumable upload)

The obvious way: send the whole 2 GB file in one go. To keep the demo small, we pretend it is 20 pieces of 100 MB (our numbers), and the train's network drops with an 8% chance on each piece (our guess). If it drops, the server throws the half-received file away, and the app starts again. Three attempts, then it gives up.

  • A chunk is one piece of a big file, cut to a fixed size. Example: a 2 GB video cut into twenty 100 MB chunks.
  • A drop is when the network connection breaks in the middle of a transfer. Python's ConnectionResetError is the error for "the other side closed the connection". Example: ConnectionResetError("upload failed, start again").
  • A traceback is Python's report of where an error happened, innermost call last.
  • random.seed(7) makes Python's random numbers repeat in the same order every run, so the output below is always the same. random.random() gives a number from 0 to 1; > 0.08 is true 92% of the time.
import random
random.seed(7)
CHUNKS = 20                    # a 2 GB video = 20 pieces of 100 MB

def network_ok():              # Priya's train goes into tunnels
    return random.random() > 0.08

def upload_whole_file():
    sent = 0
    for attempt in range(1, 4):
        for chunk in range(CHUNKS):
            sent += 100
            if not network_ok():
                print(f"attempt {attempt}: dropped at {chunk * 5}%")
                break          # the server throws the half file away
        else:
            return f"done after sending {sent:,} MB"
    print(f"sent {sent:,} MB, saved 0 MB")
    raise ConnectionResetError("upload failed, start again")

print(upload_whole_file())

Output (file paths in the traceback shortened to main.py):

attempt 1: dropped at 15%
attempt 2: dropped at 10%
attempt 3: dropped at 5%
sent 900 MB, saved 0 MB
Traceback (most recent call last):
  File "main.py", line 21, in <module>
    print(upload_whole_file())
          ~~~~~~~~~~~~~~~~~^^
  File "main.py", line 19, in upload_whole_file
    raise ConnectionResetError("upload failed, start again")
ConnectionResetError: upload failed, start again
  • Attempt 1 dropped at 15%, attempt 2 at 10%, attempt 3 at 5%. The count chunk * 5 is the position of the dropped chunk as a percent of 20 chunks.
  • The app sent 900 MB and saved 0 MB, then raised ConnectionResetError: upload failed, start again. (The video says 1,000 MB; the real run of this exact code prints 900: each attempt counts the chunk that dropped too, 400 + 300 + 200.)
  • Why? One drop throws away everything sent so far. The bigger the file, the more likely a drop happens somewhere, so a big file on a train may never finish at all. All or nothing means nothing.
  • for ... else runs the else part only when the loop was never stopped by break: here, when all 20 chunks got through.

Real life: moving house in boxes

Moving house Uploading a file
Don't move the whole house at once: pack it in numbered boxes Chunks: a file in numbered pieces
Someone ticks each box off a list when it arrives The server confirms each chunk and remembers what it has
The truck breaks down: send only the missing boxes Resumable upload: continue from where it stopped

The fix: the server remembers what arrived

Same network, same random numbers. But now the server keeps a record of every chunk it has confirmed (server_has). After a drop, the app skips chunks that are already there and sends only the rest.

  • A resumable upload is an upload that can continue after a failure, without sending again what already arrived. Example: after a drop at chunk 3, chunks 0 to 2 are not sent again.
  • A set (set()) is a Python collection that holds each value at most once and answers "is this in here?" fast. server_has.add(chunk) records a chunk; chunk in server_has asks.
import random
random.seed(7)
CHUNKS = 20                    # 20 chunks of 100 MB
server_has = set()             # chunks the server has confirmed

def network_ok():
    return random.random() > 0.08

def resumable_upload():
    sent = 0
    while len(server_has) < CHUNKS:
        for chunk in range(CHUNKS):
            if chunk in server_has:
                continue       # already there: skip it
            sent += 100
            if not network_ok():
                print(f"dropped at chunk {chunk}, resume later")
                break
            server_has.add(chunk)
    return f"done after sending {sent:,} MB"

print(resumable_upload())

Output:

dropped at chunk 3, resume later
dropped at chunk 5, resume later
dropped at chunk 6, resume later
dropped at chunk 7, resume later
dropped at chunk 17, resume later
done after sending 2,500 MB
  • Five drops this time (chunks 3, 5, 6, 7 and 17), and still: done after sending 2,500 MB for a 2,000 MB file. Only the five dropped chunks were sent twice (2,000 + 5 × 100 = 2,500).
  • Compare with Problem 1: the one-go upload sent 900 MB and saved nothing.
  • Chunks also let the app upload several pieces at the same time (general knowledge), and they are our building block for everything that follows.

You just invented chunked, resumable uploads.

Problem 2: one word changed, whole file sent (blocks and hashes)

Priya has a trip plan file. She changes one word: Agra becomes Pune. Sending the whole file again is a waste, but how does the app know which part changed?

  • A block is a fixed-size piece of a file. It is the same idea as a chunk; "block" is the word Dropbox uses. Dropbox's engineering blog says: "Every file in Dropbox is partitioned into 4MB blocks, with the final block potentially being smaller." Our demo uses 4 bytes per block, so you can see it.
  • A hash function turns any data into a short, fixed-length "fingerprint". SHA-256 is one well-known hash function (general knowledge) that gives a 256-bit (32-byte) fingerprint, usually written as 64 hex characters. The same input always gives the same fingerprint; change even one byte and the fingerprint changes completely. Dropbox: "These blocks are hashed with SHA-256 and stored."
  • Hex (hexadecimal) writes numbers with the digits 0-9 and a-f. Example: 5b4e8220.

Try the fingerprint idea first:

import hashlib

def fp(data):
    return hashlib.sha256(data).hexdigest()

print("Agra :", fp(b"Agra")[:16])
print("Agra :", fp(b"Agra")[:16], "(same input, same fingerprint)")
print("Agrb :", fp(b"Agrb")[:16], "(one letter changed)")
print("length of the full hex:", len(fp(b"Agra")), "characters =", len(fp(b"Agra")) // 2, "bytes")

Output:

Agra : 5b4e8220d7f3572a
Agra : 5b4e8220d7f3572a (same input, same fingerprint)
Agrb : 89e23ec18aa0d8ee (one letter changed)
length of the full hex: 64 characters = 32 bytes
  • Same input, same fingerprint. Agra and Agrb differ by one letter, yet their fingerprints look nothing alike. That is why comparing fingerprints is a safe way to ask "did this block change?".
  • hashlib is Python's built-in library of hash functions; .hexdigest() gives the fingerprint as hex text; [:8] keeps the first 8 characters so the output fits on a line.

Now the real demo: cut the old and new file into 4-byte blocks, hash each block, and compare.

  • range(0, len(data), BLOCK) counts 0, 4, 8, 12... so data[i:i + BLOCK] is each block in turn. b"..." is a bytes value: raw bytes instead of text.
  • enumerate(after) gives each fingerprint together with its position number.
import hashlib
BLOCK = 4                      # Dropbox uses 4 MB blocks; we use 4 bytes

def block_hashes(data):
    return [hashlib.sha256(data[i:i + BLOCK]).hexdigest()[:8]
            for i in range(0, len(data), BLOCK)]

old = b"Day 1 Goa. Day 2 Agra. Day 3 Jaipur."
new = b"Day 1 Goa. Day 2 Pune. Day 3 Jaipur."   # one word changed

before, after = block_hashes(old), block_hashes(new)
changed = [i for i, h in enumerate(after)
           if i >= len(before) or h != before[i]]
print("blocks in the file:", len(after))
print("block hashes:", after[:3], "...")
print("changed blocks:", changed)
print(f"upload {len(changed)} of {len(after)} blocks, not the whole file")

Output:

blocks in the file: 9
block hashes: ['df2441a4', '4e747581', 'f7994e1f'] ...
changed blocks: [4, 5]
upload 2 of 9 blocks, not the whole file
  • The 36-byte file has 9 blocks. Only blocks 4 and 5 differ ("Agra" sits across bytes 17 to 20, so it touches two blocks). Upload 2 of 9 blocks, not the whole file.
  • On a real 1 GB file with a small edit, that can mean sending a few megabytes instead of a gigabyte (our arithmetic with Dropbox's 4 MB block size). This is called delta sync: send only the difference.
  • This is the Dropbox idea: the file is split into blocks, hashed with SHA-256, and the server is asked about the hashes. Per the Dropbox blog, the metaserver checks whether those hashes are known, and if not, the commit comes back as "need blocks" listing the missing ones. We build that exact conversation in Problem 4.

You just invented delta sync. (But see Practice 1: what if the edit is at the start of the file?)

Problem 3: the same video, three times (deduplication)

Now Priya shares the video, and her mom and Rahul each save their own copy. Three copies of 2 GB, each in their own folder? That would triple the disks for no reason.

Real life: the library and the reading list

In a library In storage
Three students want one book: the library keeps one copy One block, one hash: stored once
Each student has a reading list that points to the shelf A file is a list of hashes: pointers to blocks
The shelf takes the space; the lists are tiny The block store (hash to bytes) takes the space
  • Deduplication (dedup) means storing identical data only once and pointing to it from everywhere it is used. Example: three people save the same video; the disks hold one copy of every block.
  • A block store is the place that keeps the blocks, looked up by their hash. Think of a Python dict mapping hash to bytes.
  • block_store.setdefault(h, block) means "store this block under h only if h is not there yet".
  • random.randbytes(1_000_000) makes 1,000,000 random bytes: a stand-in for a video. 1_000_000 is just 1000000 with underscores to read it easily.
import hashlib, random
random.seed(1)
block_store = {}               # hash -> block bytes, kept once
files = {}                     # file name -> list of block hashes

def save(name, data, size=4096):
    hashes = []
    for i in range(0, len(data), size):
        block = data[i:i + size]
        h = hashlib.sha256(block).hexdigest()
        block_store.setdefault(h, block)   # only new blocks are stored
        hashes.append(h)
    files[name] = hashes

video = random.randbytes(1_000_000)      # a 1 MB "wedding video"
save("priya/wedding.mp4", video)
save("rahul/wedding.mp4", video)         # her brother saves it too
save("mom/wedding.mp4", video)           # and her mom

stored = sum(len(b) for b in block_store.values())
print(f"files: {len(files)}, each made of {len(files['mom/wedding.mp4'])} blocks")
print(f"stored: {stored:,} bytes in {len(block_store)} unique blocks")
print(f"saved by storing each block once: {3 * len(video) - stored:,} bytes")

Output:

files: 3, each made of 245 blocks
stored: 1,000,000 bytes in 245 unique blocks
saved by storing each block once: 2,000,000 bytes
  • Three files, each made of 245 blocks (1,000,000 ÷ 4,096, rounded up). But only 245 unique blocks are stored: 1,000,000 bytes, not 3,000,000. 2,000,000 bytes saved.
  • Without the hash check (every block appended to a list), the same three saves look like this:
import hashlib, random
random.seed(1)
block_store = []                      # NO hash check: every block is kept

def save(data, size=4096):
    for i in range(0, len(data), size):
        block_store.append(data[i:i + size])

video = random.randbytes(1_000_000)
for _ in range(3):                    # Priya, Rahul, mom
    save(video)
print(f"stored: {sum(len(b) for b in block_store):,} bytes in {len(block_store)} blocks")

Output:

stored: 3,000,000 bytes in 735 blocks
  • 735 blocks and 3,000,000 bytes: three times the disks.
  • A bonus: when Rahul's app wants to upload, it first sends only the hashes. If the server says "I already have all of these", the upload finishes without sending a single block.
  • A file is just a list of hashes, so it can be rebuilt by looking up each hash. Download is the same trip backwards:
import hashlib

BLOCK = 4
block_store = {}                      # hash -> bytes
data = b"Day 1 Goa. Day 2 Agra. Day 3 Jaipur."

file_entry = []                       # the file = a list of hashes
for i in range(0, len(data), BLOCK):
    block = data[i:i + BLOCK]
    h = hashlib.sha256(block).hexdigest()[:8]
    block_store[h] = block
    file_entry.append(h)

print("file entry:", file_entry)
rebuilt = b"".join(block_store[h] for h in file_entry)
print("download gives:", rebuilt.decode())
print("same as the original:", rebuilt == data)

Output:

file entry: ['df2441a4', '4e747581', 'f7994e1f', '16b6eae4', '100bd621', 'f7994e1f', 'fb918430', 'a2de2ba8', '20bdbfd5']
download gives: Day 1 Goa. Day 2 Agra. Day 3 Jaipur.
same as the original: True
  • The file entry has 9 hashes, but f7994e1f appears twice: two blocks inside the file have identical bytes, so even a single file stores that block once.
  • b"".join(...) glues the blocks back together; rebuilt == data is True, so nothing was lost.

You just invented deduplication.

Problem 4: where does the file list live? (metadata vs blocks)

We now have two very different kinds of data:

Metadata Blocks
What File names, folders, who it is shared with, versions, each file's list of block hashes The actual bytes
Size Tiny rows Huge
Changes All the time Never: a changed block is simply a new block with a new hash
Must be Exactly right (two people must never see two different versions of a folder) Cheap and durable
Lives in A metadata database A block store (object storage)
  • Metadata is data about data. Example: for trip.txt, the name, the owner, the version number and the list of block hashes; not the text itself.
  • A database is a program that stores records and answers questions about them. SQL databases (episode 9) give you transactions: either a whole change happens or none of it does. To shard (episode 12) is to split a big database into slices, here by user, so no single machine holds everything.
  • Object storage is a service that keeps big blobs of bytes under a name and is cheap for large, rarely changed data (storage and CDN episode; general knowledge). Putting gigabytes of bytes inside database rows makes the database slow and expensive. So: SQL for metadata, object storage for blocks.

The upload becomes two steps:

  1. The app uploads the missing blocks to the block store.
  2. Then it commits the new block list to the metadata database. Until the commit, nobody sees a half-uploaded file.
  • A commit means making a change official so that others can see it, all at once. Example: commit("rahul/trip.txt", hashes).

Here is the conversation, in the style Dropbox describes (the "need blocks" reply is from their blog; the code is our own toy):

import hashlib

BLOCK = 4
block_store = {}                      # the block store: hash -> bytes
metadata = {}                         # the metadata DB: path -> list of hashes

def hashes_of(data):
    return [hashlib.sha256(data[i:i + BLOCK]).hexdigest()[:8]
            for i in range(0, len(data), BLOCK)]

def commit(path, hashes):
    missing = [h for h in hashes if h not in block_store]
    if missing:
        return ("need blocks", missing)       # the server refuses for now
    metadata[path] = hashes
    return ("committed", [])

def upload(path, data):
    hashes = hashes_of(data)
    status, missing = commit(path, hashes)
    print(f"{path}: commit -> {status}, {len(missing)} missing")
    if status == "need blocks":
        for i in range(0, len(data), BLOCK):
            h = hashlib.sha256(data[i:i + BLOCK]).hexdigest()[:8]
            if h in missing:
                block_store[h] = data[i:i + BLOCK]
        status, missing = commit(path, hashes)
        print(f"{path}: commit again -> {status}")

upload("priya/trip.txt", b"Day 1 Goa. Day 2 Agra.")
upload("rahul/trip.txt", b"Day 1 Goa. Day 2 Agra.")     # same content
print("blocks stored:", len(block_store))
print("files in metadata:", sorted(metadata))

Output:

priya/trip.txt: commit -> need blocks, 6 missing
priya/trip.txt: commit again -> committed
rahul/trip.txt: commit -> committed, 0 missing
blocks stored: 6
files in metadata: ['priya/trip.txt', 'rahul/trip.txt']
  • Priya's first commit is refused: need blocks, 6 missing. The app uploads them and commits again: committed.
  • Rahul uploads the same content: the commit is committed with 0 missing. No block is sent. Two files exist in the metadata, but 6 blocks are stored.
  • The point is the order: the server checks the hashes first, the app uploads only what is missing, and only then does the commit succeed.

You just invented the metadata database + block store split.

Problem 5: two devices, one file (versions and conflicts)

Now sync. Priya's laptop and phone both have trip.txt, version 1, which says "Goa". Both go offline. On the laptop she adds Agra. On the phone she adds Pune. Then both come online.

  • A version is one saved state of a file, numbered in order. Example: trip.txt version 2.
  • Last write wins means whichever device uploads last overwrites the others. It is simple, but silent.
  • A conflict is when two devices edited the same version of a file separately. Example: both started from version 1.
  • A conflicted copy is a second file the service creates to keep the edit that could not be merged, so nothing is lost. Example: trip (phone's conflicted copy).txt.
  • A base version is the version an edit started from. The fix: every upload says which version it was edited from, and the server accepts it only if that is still the latest.
server = {"name": "trip.txt", "version": 1, "text": "Goa"}

def upload(device, base_version, text, check=True):
    if check and base_version != server["version"]:
        return f"{device}: conflict! saved as 'trip ({device}'s conflicted copy).txt'"
    server["version"] += 1
    server["text"] = text
    return f"{device}: saved version {server['version']}"

# both devices start from version 1, then edit offline
print("last write wins:")
print(" ", upload("laptop", 1, "Goa, Agra", check=False))
print(" ", upload("phone", 1, "Goa, Pune", check=False))
print("  server text:", server["text"], "<- Agra is lost")

server.update(version=1, text="Goa")
print("with versions:")
print(" ", upload("laptop", 1, "Goa, Agra"))
print(" ", upload("phone", 1, "Goa, Pune"))

Output:

last write wins:
  laptop: saved version 2
  phone: saved version 3
  server text: Goa, Pune <- Agra is lost
with versions:
  laptop: saved version 2
  phone: conflict! saved as 'trip (phone's conflicted copy).txt'
  • Last write wins: the laptop saves version 2, the phone saves version 3, and the server text is Goa, Pune. Agra is lost, silently, and nobody is told.
  • With versions: the laptop saves version 2. The phone was edited from version 1, which is no longer the latest, so it is a conflict, and the phone's edit is kept as trip (phone's conflicted copy).txt.
  • Nothing is lost, and Priya decides which to keep. Services like Dropbox and Google Drive handle this in similar ways, and keep old versions too (general knowledge; we did not fetch their exact rules).
  • f"...{device}'s..." is an f-string: the {...} parts are filled in with values. check=False switches the version check off, to show the bad way.

You just invented versioning and conflict handling.

Problem 6: asking every 5 seconds (a change log and a push)

Last problem. How does the laptop learn that the phone changed something? The obvious way: ask the server every 5 seconds, "anything new?".

  • Polling means asking again and again whether something has changed. Example: the laptop sends "anything new?" every 5 seconds.
  • A change log is a list, kept by the server for each user, of every change in order. Example: v1: made trip.txt, v2: added beach.jpg.
  • A cursor is a number a device remembers: how far down the change log it has already read. The laptop asks only for entries after its cursor.
  • A push (here) is the server telling the device "something changed" over a connection the device keeps open, instead of the device asking. A long poll is a request the server holds open until it has news; a WebSocket is a connection that stays open so either side can send at any time (general knowledge).

Our guesses: 10 million devices online, about 20 real changes per device per day.

  • 86_400 is the number of seconds in a day (24 × 60 × 60).
  • {asks:,.0f} prints a number with commas and no decimals; {x:.2%} prints a fraction as a percent with 2 decimals.
  • log[laptop_cursor:] is a slice: the entries from position 1 to the end.
devices = 10_000_000           # devices online (our guess)
poll_every = 5                 # seconds between "anything new?"
changes = 20                   # real changes per device a day (guess)

asks = devices * 86_400 / poll_every
useful = devices * changes
print(f"polling: {asks:,.0f} requests a day")
print(f"useful:  {useful:,.0f} of them ({useful / asks:.2%})")

# the fix: a change log, and each device remembers a cursor
log = ["v1: made trip.txt", "v2: added beach.jpg", "v3: edited trip.txt"]
laptop_cursor = 1              # the laptop has seen entry 0 only
print("server pushes: 'something changed'")
print("laptop asks for log[1:] ->", log[laptop_cursor:])

Output:

polling: 172,800,000,000 requests a day
useful:  200,000,000 of them (0.12%)
server pushes: 'something changed'
laptop asks for log[1:] -> ['v2: added beach.jpg', 'v3: edited trip.txt']
  • 172,800,000,000 requests a day (172.8 billion), and only 200,000,000 of them (0.12%) find anything new. The rest are wasted.
  • The fix has two parts. The server keeps a change log; each device remembers its cursor. The laptop has seen entry 0 only, so it asks for log[1:] and gets exactly the two entries it missed.
  • Instead of asking again and again, the device keeps one connection open (a long poll or a WebSocket) and the server pushes "something changed". Then the device fetches log[cursor:]. Don't ask, get told.

You just invented efficient sync.

Problem 7: the scale

How much disk does this need? Say 50 million users, storing 5 GB each on average, and 30% of blocks are duplicates. Every block is kept 3 times for durability (a replica is one more copy of the same data, episode 11). All of these are our assumptions.

users = 50_000_000             # users (our assumption)
per_user_gb = 5                # stored per user, on average (guess)
replicas = 3                   # every block kept 3 times (episode 11)
dedup = 0.7                    # 30% of blocks are duplicates (guess)

raw_pb = users * per_user_gb / 1e6
print(f"what users store: {raw_pb:,.0f} PB")
print(f"after dedup:      {raw_pb * dedup:,.0f} PB")
print(f"with 3 copies:    {raw_pb * dedup * replicas:,.0f} PB of disks")

Output:

what users store: 250 PB
after dedup:      175 PB
with 3 copies:    525 PB of disks
  • 50,000,000 × 5 GB = 250,000,000 GB = 250 PB of user data.
  • Dedup keeps 70%: 175 PB. Three copies of every block: 525 PB of disks.
  • That is why dedup matters so much, and why old, rarely opened blocks are usually moved to cheaper cold storage (storage that is cheap to keep but slower to read; general knowledge).

The full design

  client app                API servers              metadata DB               block store
  ──────────                ───────────              ───────────               ───────────
  watch folder             login, auth               files, folders,           hash → bytes
  cut into blocks  ─hashes▶ commit(path, base,  ────▶ versions, shares  ───▶   3 copies each
  hash each block           hashes)                  change log per user       cold tier for old blocks
  upload missing blocks ───────────────────────────────────────────────────▶
        ▲                        │ conflict? → conflicted copy
        │                        ▼
        └──── "something changed" push ◀── notification service (long poll / WebSocket)
  downloads of popular shared files ◀── CDN
  1. The client app watches a folder, cuts files into blocks and hashes them.
  2. It sends the hashes to the API servers; the server answers "need blocks" with the missing ones. The app uploads only those, to the block store.
  3. The app then commits the new block list to the metadata database. The commit checks the base version: if it is stale, the result is a conflicted copy.
  4. Every commit is appended to the user's change log, and a notification service tells the user's other devices; each fetches log[cursor:].
  5. Downloads of popular shared files can come from a CDN (a network of servers near users that caches popular files), and old blocks move to cold storage.
  6. The idea that holds it together: a file is just a list of block hashes.

(The pieces are the episode's design. Which of them a real service uses, and how, is not something our sources say; Dropbox's blog confirms only the 4 MB blocks, SHA-256 and the "need blocks" check.)

The whole chain

  1. A tunnel threw away the upload (900 MB sent, 0 saved) → chunks + resumable upload.
  2. One changed word re-sent the whole file → block hashes, 2 of 9 blocks sent.
  3. The same video × 3 → each block stored once (1,000,000 bytes, not 3,000,000).
  4. Bytes in database rows are slow → metadata DB + block store, blocks first then commit.
  5. "Agra" lost by last write wins → versions + a conflicted copy.
  6. 0.12% of polls useful → change log + cursor + a push.
  7. 250 PB of user data → 525 PB of disks → dedup, cold storage.

Common mistakes (and the real outputs)

Mistake What you see Why Fix
Upload the whole file in one request ConnectionResetError: upload failed, start again One drop throws away everything sent so far Chunks the server confirms; resume
Save whatever arrives last server text: Goa, Pune <- Agra is lost Last write wins hides conflicts Check the base version; make a conflicted copy
Ask the server every 5 seconds useful: 200,000,000 of them (0.12%) Almost every poll finds nothing Change log + cursor + a push
Store every block without checking its hash stored: 3,000,000 bytes in 735 blocks Three copies of the same video block_store.setdefault(h, block)
Fixed-size blocks, edit at the start upload 10 of 10 blocks, not the whole file (Practice 1) Every byte shifts, so every block changes Content-defined chunking (general knowledge)
Hash a str instead of bytes TypeError: Strings must be encoded before hashing hashlib hashes bytes, not text "Agra".encode() or b"Agra"
Block size 0 ValueError: range() arg 3 must not be zero range cannot step by 0 Use a block size of at least 1
Read a block that was never uploaded KeyError: '4e747581' The commit pointed to a hash that is not in the store Commit only after the "need blocks" list is empty

The real tracebacks (paths shortened to main.py):

import hashlib
print(hashlib.sha256("Agra").hexdigest())
Traceback (most recent call last):
  File "main.py", line 2, in <module>
    print(hashlib.sha256("Agra").hexdigest())
          ~~~~~~~~~~~~~~^^^^^^^^
TypeError: Strings must be encoded before hashing
BLOCK = 0
data = b"Day 1 Goa."
for i in range(0, len(data), BLOCK):
    print(data[i:i + BLOCK])
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    for i in range(0, len(data), BLOCK):
             ~~~~~^^^^^^^^^^^^^^^^^^^^^
ValueError: range() arg 3 must not be zero
block_store = {"df2441a4": b"Day "}
print(block_store["4e747581"])
Traceback (most recent call last):
  File "main.py", line 2, in <module>
    print(block_store["4e747581"])
          ~~~~~~~~~~~^^^^^^^^^^^^
KeyError: '4e747581'

Mini project: a tiny file storage service

A server with a block store, a metadata table with versions, a change log; devices that cut files into 16-byte blocks, ask the server which blocks it needs, resume after network drops, get conflicted copies, and pull changes from a cursor. The phone's network dies on its 3rd and 7th block upload. Rahul saves the same video as Priya, then the laptop edits one block of it, then the laptop and phone edit trip.txt from the same version.

# Mini project: a tiny file storage service. Blocks + hashes, resumable upload
# (the server says "need blocks"), dedup, versions, conflicted copies and a
# change log with a cursor per device. Everything runs in memory.
import hashlib, random

BLOCK = 16                                   # Dropbox: 4 MB; ours: 16 bytes

def sha(piece):
    return hashlib.sha256(piece).hexdigest()[:8]

class Server:
    def __init__(self):
        self.blocks = {}                     # block store: hash -> bytes
        self.files = {}                      # metadata DB: path -> [block lists]
        self.log = []                        # change log: (path, version, device)

    def commit(self, path, base, hashes, device):
        missing = [h for h in dict.fromkeys(hashes) if h not in self.blocks]
        if missing:
            return "need blocks", missing    # nothing is visible yet
        versions = self.files.setdefault(path, [])
        if base != len(versions):            # edited from an old version
            stem, dot, ext = path.rpartition(".")
            copy = f"{stem} ({device}'s conflicted copy).{ext}"
            self.files[copy] = [hashes]
            self.log.append((copy, 1, device))
            return "conflict", copy
        versions.append(hashes)              # old versions stay
        self.log.append((path, len(versions), device))
        return "ok", len(versions)

class Device:
    def __init__(self, name, server, drops=()):
        self.name, self.server = name, server
        self.drops, self.tries = set(drops), 0   # block uploads that die
        self.seen, self.cursor = {}, 0       # path -> version, log position
        self.sent = self.logical = 0         # bytes uploaded / bytes saved

    def save(self, path, data):
        pieces = [data[i:i + BLOCK] for i in range(0, len(data), BLOCK)]
        hashes = [sha(p) for p in pieces]
        by_hash = dict(zip(hashes, pieces))
        self.logical += len(data)
        while True:
            status, info = self.server.commit(path, self.seen.get(path, 0),
                                              hashes, self.name)
            if status != "need blocks":
                break
            print(f"  {self.name}: server needs {len(info)} of {len(hashes)} blocks")
            for h in info:
                self.tries += 1
                if self.tries in self.drops:
                    print(f"  {self.name}: network dropped, asking again")
                    break
                self.server.blocks[h] = by_hash[h]
                self.sent += len(by_hash[h])
        if status == "ok":
            self.seen[path] = info
            print(f"{self.name}: {path} saved as version {info}")
        else:
            print(f"{self.name}: conflict! saved as '{info}'")

    def pull(self):
        news = self.server.log[self.cursor:]
        self.cursor = len(self.server.log)
        for path, version, device in news:
            self.seen[path] = version
        print(f"{self.name}: pulled {len(news)} changes, newest: {news[-1][:2]}")

server = Server()
phone = Device("phone", server, drops=[3, 7])    # a train: 2 drops
laptop = Device("laptop", server)
rahul = Device("rahul", server)

video = random.Random(3).randbytes(160)          # 10 blocks
phone.save("priya/wedding.mp4", video)
rahul.save("rahul/wedding.mp4", video)           # same bytes
edited = video[:64] + b"X" * 16 + video[80:]     # block 4 replaced
laptop.pull()                                    # the laptop catches up
laptop.save("priya/wedding.mp4", edited)

phone.save("priya/trip.txt", b"Goa")
laptop.pull()
laptop.save("priya/trip.txt", b"Goa, Agra")      # offline edit 1
phone.save("priya/trip.txt", b"Goa, Pune")       # offline edit 2, same base
phone.pull()

stored = sum(len(b) for b in server.blocks.values())
sent = phone.sent + laptop.sent + rahul.sent
logical = phone.logical + laptop.logical + rahul.logical
print(f"bytes the users saved: {logical}, bytes sent: {sent}, bytes stored: {stored}")
print("versions of priya/wedding.mp4:", len(server.files["priya/wedding.mp4"]))
print("files:", sorted(server.files))

Output:

  phone: server needs 10 of 10 blocks
  phone: network dropped, asking again
  phone: server needs 8 of 10 blocks
  phone: network dropped, asking again
  phone: server needs 5 of 10 blocks
phone: priya/wedding.mp4 saved as version 1
rahul: rahul/wedding.mp4 saved as version 1
laptop: pulled 2 changes, newest: ('rahul/wedding.mp4', 1)
  laptop: server needs 1 of 10 blocks
laptop: priya/wedding.mp4 saved as version 2
  phone: server needs 1 of 1 blocks
phone: priya/trip.txt saved as version 1
laptop: pulled 2 changes, newest: ('priya/trip.txt', 1)
  laptop: server needs 1 of 1 blocks
laptop: priya/trip.txt saved as version 2
  phone: server needs 1 of 1 blocks
phone: conflict! saved as 'priya/trip (phone's conflicted copy).txt'
phone: pulled 6 changes, newest: ("priya/trip (phone's conflicted copy).txt", 1)
bytes the users saved: 501, bytes sent: 197, bytes stored: 197
versions of priya/wedding.mp4: 2
files: ["priya/trip (phone's conflicted copy).txt", 'priya/trip.txt', 'priya/wedding.mp4', 'rahul/wedding.mp4']
  • Resume: the phone's upload of the 10-block video needed 10, then 8, then 5 blocks: after each drop it asked again and only the missing ones were sent. Priya's phone sent each block once.
  • Dedup: Rahul saved the same bytes: no "server needs" line at all, because every hash was known.
  • Delta sync: the laptop's edit changed one block of ten, so "server needs 1 of 10 blocks", and the file went to version 2 (version 1 is kept).
  • Conflict: the laptop saved trip.txt version 2; the phone, still on version 1, got a conflicted copy. The phone's block was uploaded before the version check (blocks first, then commit), so it stays in the block store.
  • Change log: the laptop pulled 2 changes, then 2 more; the phone, which had never pulled, pulled all 6 in one go from cursor 0.
  • Numbers: users saved 501 bytes in total, but only 197 were sent and 197 stored.

Try extending it: add a third device and see its conflicts; make the block store count how many files point to each block, and delete a block only when the count is 0; add a quota per user; split the files into folders.

Practice

  1. Homework: in the BLOCKS demo, insert a word at the start of the file instead of changing one. What breaks?
  2. Homework: in the RESUME demo, try a 20% drop chance. How many MB are sent? What does the one-go upload do?
  3. Homework: in the CONFLICT demo, add a third device (a tablet).
  4. Homework (comment): what should happen when someone deletes a shared file?
  5. In the SCALE demo, only 50% of blocks are duplicates. What is the disk total?
  6. In the POLL demo, poll every 60 seconds instead of 5. Is polling now fine?
  7. Two different blocks produce the same SHA-256 fingerprint. What would go wrong in a dedup store, and should you worry?
  8. Using Google's two limits (5 TB per file, 750 GB per user per 24 hours) and Dropbox's 4 MB blocks: how many 2 GB videos fit in a day's upload, and how many 4 MB blocks does a 5 TB file have?
Solution

1. Change new to start with a word, new = b"Hi " + old. We also print the blocks the server has never seen:

import hashlib
BLOCK = 4                      # Dropbox uses 4 MB blocks; we use 4 bytes

def block_hashes(data):
    return [hashlib.sha256(data[i:i + BLOCK]).hexdigest()[:8]
            for i in range(0, len(data), BLOCK)]

old = b"Day 1 Goa. Day 2 Agra. Day 3 Jaipur."
new = b"Hi " + old                # a word inserted at the START

before, after = block_hashes(old), block_hashes(new)
changed = [i for i, h in enumerate(after)
           if i >= len(before) or h != before[i]]
unknown = [i for i, h in enumerate(after) if h not in before]
print("blocks in the file:", len(after))
print("changed blocks:", changed)
print("blocks the server has never seen:", unknown)
print(f"upload {len(changed)} of {len(after)} blocks, not the whole file")

Output:

blocks in the file: 10
changed blocks: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
blocks the server has never seen: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
upload 10 of 10 blocks, not the whole file

Inserting 3 bytes at the front shifts every later byte by 3, so every 4-byte block now holds different bytes, and all 10 blocks changed. The whole file is uploaded again, which is exactly what delta sync was meant to avoid. (The last line of the demo says "10 of 10 blocks, not the whole file", which is now just the whole file: the message is wrong for this case.) A fixed-size block cuts by position, so it cannot survive a shift.

The idea that fixes it is content-defined chunking (general knowledge, not from our sources): instead of cutting every N bytes, cut where the content says so, for example where a rolling hash of the last few bytes hits a special value, so the cut points move with the data. Here is a toy version that cuts after every ". ":

import hashlib

def pieces(data):                      # cut AFTER every ". " (content decides)
    parts = data.split(b". ")
    return [p + b". " for p in parts[:-1]] + [parts[-1]]

def hashes(data):
    return [hashlib.sha256(p).hexdigest()[:8] for p in pieces(data)]

old = b"Day 1 Goa. Day 2 Agra. Day 3 Jaipur."
new = b"Hi " + old                       # insert at the start again
before, after = hashes(old), hashes(new)
print("pieces:", [p.decode() for p in pieces(new)])
print("pieces the server has never seen:", [i for i, h in enumerate(after) if h not in before])

Output:

pieces: ['Hi Day 1 Goa. ', 'Day 2 Agra. ', 'Day 3 Jaipur.']
pieces the server has never seen: [0]

Now the inserted word changes only piece 0; the other two pieces are identical, so only one piece needs uploading. (A real system cuts by a rolling hash of the bytes, not by a full stop, so it works on videos too. We did not check what Dropbox or Drive actually do beyond Dropbox's fixed 4 MB blocks.)

2. Change > 0.08 to > 0.20 in network_ok. Output:

dropped at chunk 1, resume later
dropped at chunk 2, resume later
dropped at chunk 4, resume later
dropped at chunk 5, resume later
dropped at chunk 6, resume later
dropped at chunk 6, resume later
dropped at chunk 8, resume later
dropped at chunk 14, resume later
dropped at chunk 16, resume later
dropped at chunk 16, resume later
dropped at chunk 18, resume later
done after sending 3,100 MB

The resumable upload still finishes, but after 11 drops it has sent 3,100 MB (2,000 + 11 × 100) for the 2,000 MB file. Now the same 20% change in the one-go upload (WHOLE):

attempt 1: dropped at 5%
attempt 2: dropped at 5%
attempt 3: dropped at 10%
sent 700 MB, saved 0 MB
Traceback (most recent call last):
  File "main.py", line 21, in <module>
    print(upload_whole_file())
          ~~~~~~~~~~~~~~~~~^^
  File "main.py", line 19, in upload_whole_file
    raise ConnectionResetError("upload failed, start again")
ConnectionResetError: upload failed, start again

It fails again, with an even worse score: 700 MB sent, 0 saved. At 20% per chunk, a 20-chunk file almost never gets through in one go (0.8^20 is about 1%, our arithmetic), while the resumable one just keeps going.

3. Add a tablet:

server = {"name": "trip.txt", "version": 1, "text": "Goa"}

def upload(device, base_version, text, check=True):
    if check and base_version != server["version"]:
        return f"{device}: conflict! saved as 'trip ({device}'s conflicted copy).txt'"
    server["version"] += 1
    server["text"] = text
    return f"{device}: saved version {server['version']}"

# three devices start from version 1, then edit offline
print("last write wins:")
print(" ", upload("laptop", 1, "Goa, Agra", check=False))
print(" ", upload("phone", 1, "Goa, Pune", check=False))
print(" ", upload("tablet", 1, "Goa, Jaipur", check=False))
print("  server text:", server["text"], "<- Agra and Pune are lost")

server.update(version=1, text="Goa")
print("with versions:")
print(" ", upload("laptop", 1, "Goa, Agra"))
print(" ", upload("phone", 1, "Goa, Pune"))
print(" ", upload("tablet", 1, "Goa, Jaipur"))

Output:

last write wins:
  laptop: saved version 2
  phone: saved version 3
  tablet: saved version 4
  server text: Goa, Jaipur <- Agra and Pune are lost
with versions:
  laptop: saved version 2
  phone: conflict! saved as 'trip (phone's conflicted copy).txt'
  tablet: conflict! saved as 'trip (tablet's conflicted copy).txt'

With last write wins, the server ends with Goa, Jaipur; Agra and Pune are both lost. With versions, the laptop wins the race to version 2, and both the phone and the tablet (still on version 1) get their own conflicted copy. Nothing is lost; Priya merges by hand.

4. There is no single right answer; good ideas to mention: if the owner deletes, the others may lose access (or keep a copy if they "added it to their Drive"); a trash / recycle bin so a delete can be undone for some days; deleting means removing the metadata entry first, and a block can only be freed when no file points to it any more (dedup means one block may be shared by many files, so keep a count per block). These are design choices and general knowledge, not facts from our sources.

5. Change dedup = 0.7 to dedup = 0.5. Output:

what users store: 250 PB
after dedup:      125 PB
with 3 copies:    375 PB of disks

125 PB after dedup and 375 PB of disks, instead of 525 PB. The more duplicates, the more dedup saves. The 30% and 50% are both our guesses; the real fraction depends on the data.

6. Change poll_every = 5 to poll_every = 60. Output:

polling: 14,400,000,000 requests a day
useful:  200,000,000 of them (1.39%)
server pushes: 'something changed'
laptop asks for log[1:] -> ['v2: added beach.jpg', 'v3: edited trip.txt']

Requests drop to 14.4 billion a day, but still only 1.39% find anything, and now a change shows up on the laptop up to a minute late. Polling trades delay against waste. A push gives both: no wasted asks and news at once.

7. That is called a hash collision. In a dedup store the second block would be thrown away as "already stored", and the file would later be rebuilt with the wrong bytes, silently. For SHA-256 this is considered astronomically unlikely, which is why dedup stores can trust the hash (general knowledge; a weak hash would be a real risk). That is also why the design uses a strong hash like SHA-256, as Dropbox does, and not a short checksum.

8. Run:

video_gb = 2
daily_gb = 750                       # Drive: per user, upload and copy, per 24 hours
print(f"2 GB videos that fit in 750 GB: {daily_gb // video_gb}")

max_file_mb = 5_000_000              # 5 TB, counting 1 TB = 1,000,000 MB
block_mb = 4
blocks = max_file_mb // block_mb
print(f"4 MB blocks in a 5 TB file: {blocks:,}")
hash_bytes = 32                      # a SHA-256 fingerprint is 32 bytes
print(f"block list for it: {blocks * hash_bytes / 1e6:,.0f} MB of hashes")

Output:

2 GB videos that fit in 750 GB: 375
4 MB blocks in a 5 TB file: 1,250,000
block list for it: 40 MB of hashes

750 ÷ 2 = 375 videos of 2 GB in a day. A 5 TB file (counted as 5,000,000 MB, our simplification) is 1,250,000 blocks of 4 MB. At 32 bytes per SHA-256 fingerprint that block list is only about 40 MB of hashes for a 5 TB file: the metadata stays tiny compared with the data, which is why it fits in a database while the blocks go in a block store.

Quick quiz

client: hashes = [a91f, 07c2, ... 245 of them]
server: I already have all 245
client: upload ???

Rahul saves a video the server already has. What gets uploaded?

  • A. The whole video again
  • B. Only the first block
  • C. Nothing: just commit the block list
  • D. A zip of the video
Answer

C. Every block's hash is already in the block store, so the app only commits a new file entry pointing to those blocks. That is dedup making an upload instant.

Interview questions: "Design Google Drive / Dropbox"

1. Clarify requirements. Upload, download, sync across devices, share, version history? How big can a file be (Google's help page says up to 5 TB)? How many users and how much data each? Offline edits? Do we need real-time sync or is a few seconds fine? Then state: durable, works on bad networks, sends and stores as little as possible.

2. Estimate. "50 million users × 5 GB (our assumptions) is 250 PB; with 30% dedup 175 PB; three copies 525 PB of disks. A 4 MB block size means a 5 TB file has about 1.25 million blocks, and its block list is about 40 MB of 32-byte hashes."

3. Components.

  • Client app: watches folders, cuts files into blocks, hashes them, uploads only what the server needs, resumes after drops.
  • API servers: login, commit, "need blocks" check, version check.
  • Metadata database: names, folders, versions, shares and each version's block list; sharded by user.
  • Block store: hash to bytes, object storage, replicated; dedup because the key is the hash.
  • Change log + notification service: per-user log, device cursors, a long-poll or WebSocket push.
  • CDN for popular shared downloads, cold storage for old blocks.

4. The hard parts. Resumable uploads on unreliable networks; changed-block detection (and the shifting-insert problem of fixed blocks); dedup and the reference count that decides when a block can be deleted; atomic commit (blocks first, then metadata); conflicts between offline devices; telling devices about changes without constant polling.

5. Scale and reliability (general knowledge). Shard the metadata DB by user id; replicate blocks across machines or zones; move cold blocks to cheaper storage; rate-limit per user (Google caps uploads per user per 24 hours); verify block hashes on upload and periodically on disk to catch corruption; keep the change log durable so a device can always catch up from its cursor.

Common follow-ups:

  • What if the app crashes halfway through an upload? (The blocks already stored stay; on restart the app asks the server "need blocks" again and sends only the rest.)
  • Two devices edit the same file offline? (The commit carries the base version; the stale one becomes a conflicted copy; nothing is lost.)
  • How do you delete a block safely? (Only when no file version points to it: keep a reference count or run a garbage collector.)
  • Why not store file bytes in the database? (Slow and costly; metadata and blocks have very different shapes.)
  • Why not poll? (172.8 billion asks a day to find 200 million changes; use a cursor and a push.)

Glossary

Term Meaning
File storage service A service that keeps your files on the company's servers so you can open and share them from any device
Server / client The service's computer that answers requests / the app on your device that sends them
Upload / download Sending a file to the server / getting it from the server
Sync Keeping the same files up to date on all your devices
Share Letting other people open your file
Offline A device with no internet connection right now
Byte, KB, MB, GB, TB, PB The smallest counted unit of data, and units 1,000 times bigger each step (rough rule)
Durability Data survives crashes and broken disks
Chunk A piece of a big file, cut to a fixed size, e.g. 100 MB
Drop The network connection breaking in the middle of a transfer
ConnectionResetError Python's error for a connection closed by the other side
Traceback Python's report of where an error happened, innermost call last
Resumable upload An upload that continues after a failure without re-sending what arrived
Set Python collection holding each value at most once, with a fast "is it in here?"
Block A fixed-size piece of a file; Dropbox uses 4 MB blocks
Hash function / SHA-256 Turns any data into a fixed-length fingerprint; SHA-256 gives 32 bytes (64 hex characters)
Fingerprint (hash) The short result of a hash function; same input, same fingerprint
Hex Numbers written with 0-9 and a-f
hashlib Python's built-in library of hash functions
bytes (b"...") Raw bytes in Python, as opposed to text
Delta sync Sending only what changed
Deduplication (dedup) Storing identical data once and pointing to it from everywhere it is used
Block store The place that keeps blocks, looked up by hash (hash to bytes)
Metadata Data about data: names, folders, versions, sharing, each file's block list
Metadata database The database holding metadata; SQL, sharded by user
Metaserver Dropbox's name for the server that checks hashes and takes commits
Database / SQL / transaction A program that stores records / the common query language / a change that happens fully or not at all
Shard One slice of a big database, split e.g. by user
Object storage Cheap storage for big blobs of bytes under a name
Commit Making a change official so that others can see it all at once
"Need blocks" The server's reply listing the block hashes it does not have yet
Version One saved state of a file, numbered in order
Base version The version an edit started from
Last write wins Whoever saves last overwrites the others, silently
Conflict Two devices edited the same version separately
Conflicted copy A second file keeping the edit that could not be merged
Polling Asking again and again whether something changed
Change log A per-user list of every change, in order
Cursor How far down the change log a device has read
Push The server tells the device something changed
Long poll / WebSocket A request held open until news arrives / a connection that stays open both ways
Replica One extra copy of the same data (here: every block kept 3 times)
CDN A network of servers near users that caches popular files
Cold storage Cheap, slower storage for rarely read data
Hash collision Two different inputs with the same fingerprint; astronomically unlikely with SHA-256
Content-defined chunking Cutting a file where its content says so (not every N bytes), so an insert does not shift every block
Rolling hash A hash over a sliding window of bytes, used to find cut points (general knowledge)
seed Makes Python's random numbers repeat in the same order every run
f-string A Python string with {...} parts filled in with values
Slice (log[1:]) The part of a list from position 1 to the end
Quota / limit A cap on how much a user may store or upload

Summary

  • A file storage service must be durable, work on bad networks, and store as little as possible. Real limits exist: Google Workspace allows files up to 5 TB and 750 GB of upload per user per 24 hours.
  • Cut files into chunks the server confirms: after a drop, send only the missing ones (the one-go upload sent 900 MB and saved 0; the resumable one finished after 2,500 MB for a 2,000 MB file).
  • Cut files into blocks hashed with SHA-256: compare fingerprints and upload only the changed blocks (2 of 9). Dropbox: 4 MB blocks, SHA-256, a "need blocks" reply. Fixed-size blocks break when you insert at the start (10 of 10); content-defined chunking is the idea that fixes it.
  • Dedup: a block store keyed by hash stores each block once (1,000,000 bytes, not 3,000,000); a file is a list of hashes.
  • Metadata DB (SQL, sharded) for names, versions and block lists; block store (object storage) for bytes. Upload blocks first, then commit.
  • Versions + base version: no silent overwrite; a stale edit becomes a conflicted copy.
  • Change log + cursor + a push instead of polling (0.12% of polls useful).
  • 50 million users × 5 GB = 250 PB, 175 PB after dedup, 525 PB of disks with 3 copies.

Next episode (39): the system design interview framework: how to take a big question like this one and answer it in 45 minutes, step by step.

Sources: Dropbox tech blog, "Streaming File Synchronization" (https://dropbox.tech/infrastructure/streaming-file-synchronization, fetched 6 Nov 2026): "Every file in Dropbox is partitioned into 4MB blocks, with the final block potentially being smaller."; "These blocks are hashed with SHA-256 and stored."; the metaserver checks whether those hashes are known, and if not, the commit returns "need blocks" listing the missing ones. Google Workspace Admin Help, "Storage and upload limits for Google Workspace" (https://knowledge.workspace.google.com/admin/drive/storage-and-upload-limits-for-google-workspace, fetched 6 Nov 2026): "You can upload and synchronize files up to 5 TB in size."; "Each user can upload and copy 750 GB to Drive within 24 hours." Our own stated assumptions: the 8% drop chance, 100 MB chunks, 4-byte and 16-byte demo blocks, 50 million users with 5 GB each, 30% (and 50%) duplicates, 3 copies, 10 million devices polling every 5 s (and every 60 s), 20 changes a day, the 5 TB = 5,000,000 MB simplification, and all names. What chunking, hashing, SHA-256 in general, hash collisions, content-defined chunking and rolling hashes, deduplication, metadata vs object storage, conflicted copies, change logs and cursors, long polls and WebSockets, CDNs and cold storage are is general textbook knowledge; we did not fetch how Google Drive or Dropbox implement conflicts, sync notifications or storage internally.