SyntaxPilot
Courses/Cybersecurity Fundamentals/Lesson 47
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Cybersecurity Fundamentals · Lesson 47 of 90

File uploads, path traversal and insecure deserialisation

unrestricted file uploadcontent-type and extension validationmagic bytesstoring uploads outside the webrootrenaming filesmalware scanningpath traversal (../)canonicalisationzip slipinsecure deserialisationgadget chains (concept)Python pickleJava serializationsafe data formats (JSON with schemas)

These are the notes for Cybersecurity Fundamentals (defensive), episode 47. Episode 46 hardened the settings. Today we look at three places where an app takes something from a user and trusts it too much: a file (uploads), a file name (path traversal and zip slip) and a saved object (deserialisation). Every demo runs live, every fix is shown, and every term is explained the first time it appears; the Glossary at the end defines them all again.

Neha, Ravi, the shop and every value are made up. Every folder and archive is a toy the program creates in a temporary folder, the "secret" files are toy files the program writes itself, and nothing outside the machine is contacted. The harmful-looking demos only ever call print(). Python samples run on 12 Nov 2026 with Python 3.13, the Java sample with OpenJDK 21; every output below is copied from a real run. Random values (file names, temporary paths) differ each run.

The problems you'll solve

Problem Idea it teaches
A "profile photo" that is really a web page unrestricted file upload, Content-Type, MIME type, extension
Labels that lie about a file extension allowlist, magic bytes (file signatures), size limits
Uploads saved under the user's name in the web folder random file names, storing outside the webroot, malware scanning
"Download your invoice" reads a password file path traversal (../), absolute paths, realpath + commonpath
Blocking ../ is bypassed canonicalisation, URL encoding, strip-once filters
A zip that writes outside its folder archives, zip slip, safe extraction, tarfile filters
A cookie that makes the server run code serialisation, deserialisation, pickle, __reduce__, find_class
Attacks made of your own classes gadget chains (concept), Java serialization, ObjectInputFilter
The real fix safe data formats: JSON with a schema

Problem 1: a file upload that trusts the label

Basic words first:

  • Uploading a file means sending it from your device to a server, for example a profile photo or an invoice.
  • When a browser uploads a file it sends three things the user controls: the file's name (neha.png), the file's bytes, and a Content-Type.
  • The Content-Type (also called a MIME type) is a label saying what kind of data this claims to be, such as image/png, image/jpeg or application/pdf. It is just text in the request, so the uploader can write anything there. OWASP's File Upload Cheat Sheet: "don't trust the Content-Type header as it can be spoofed" (spoofed = faked).
  • The extension is the end of the file name after the last dot (.png). It is also chosen by the uploader.
  • The webroot is the folder a web server shows to the world: anything saved there can be opened by anyone with its address.
  • Unrestricted file upload means accepting a file without checking what it really is, what it is called, or where it goes.

Real-life example: a courier counter. The sender writes the label ("documents"). A careful counter X-rays the parcel (magic bytes), gives it its own tracking number instead of the sender's name (our own file name), and keeps parcels in the back room, not the shop window (outside the webroot).

Demo 1: an upload that trusts the label

# Upload v1: trust the browser's Content-Type and the file's own name
import os, tempfile
WEBROOT = tempfile.mkdtemp()            # files here are served to everyone

def upload_v1(filename, content_type, data):
    if content_type not in ("image/png", "image/jpeg"):
        return "rejected: not an image"
    path = os.path.join(WEBROOT, filename)
    open(path, "wb").write(data)
    return "saved " + filename + " in the web folder"

# Neha uploads a real photo
png = b"\x89PNG\r\n\x1a\n" + b"...pixels..."
print(upload_v1("neha.png", "image/png", png))

# someone else: a web page, with a header that *claims* image/png
page = b"<html><script>steal_cookies()</script></html>"
print(upload_v1("profile.html", "image/png", page))

# and a name that replaces the site's own home page
print(upload_v1("index.html", "image/jpeg", page))
print("web folder now:", sorted(os.listdir(WEBROOT)))

Real output:

saved neha.png in the web folder
saved profile.html in the web folder
saved index.html in the web folder
web folder now: ['index.html', 'neha.png', 'profile.html']

Why it failed: the label and the name both came from the user, so the user decided everything. A page with a script, served from our own domain, can run in every visitor's browser (cross-site scripting, episode 40), and index.html replaced our own home page.

Problem 1b: check what the file IS

  • An allowlist is a list of the only things accepted; everything else is refused. For uploads: "List allowed extensions" (OWASP), for example only .png, .jpg, .pdf. A blocklist (list of bad extensions) always misses one.
  • Magic bytes (a file signature) are the first few bytes of a file that mark its real format, whatever its name says:
Format First bytes (hex) As text
PNG 89 50 4E 47 0D 0A 1A 0A \x89PNG\r\n\x1a\n
JPEG FF D8 FF
PDF 25 50 44 46 2D %PDF-
GIF 47 49 46 38 37 61 / ...39 61 GIF87a / GIF89a

OWASP adds that a signature check "should not be used on its own": a file can begin with the right bytes and still carry something harmful later, so it is one layer among several.

  • A size limit stops one upload from filling the disk ("Set a file size limit").
  • Renaming: "Change the filename to something generated by the application". We pick a random name with secrets.token_hex, so the user cannot choose a name that replaces a file, contains odd characters or climbs out of the folder. Keep the original name in the database if you need to show it.

Demo 2: a safer upload

# Upload v2: check what the file IS, then store it safely
import os, secrets, tempfile
STORE = tempfile.mkdtemp()              # outside the web folder, never served
ALLOWED = {".png": b"\x89PNG\r\n\x1a\n", ".jpg": b"\xff\xd8\xff",
           ".pdf": b"%PDF-"}             # extension -> its magic bytes
MAX_BYTES = 2_000_000

def upload_v2(filename, data):
    ext = os.path.splitext(filename)[1].lower()
    if ext not in ALLOWED:
        return f"rejected {filename}: extension {ext or 'none'} not allowed"
    if not data.startswith(ALLOWED[ext]):
        return f"rejected {filename}: content is not really {ext}"
    if len(data) > MAX_BYTES:
        return f"rejected {filename}: too big"
    new_name = secrets.token_hex(8) + ext            # we choose the name
    open(os.path.join(STORE, new_name), "wb").write(data)
    return f"saved {filename} as {new_name}"

png = b"\x89PNG\r\n\x1a\n" + b"...pixels..."
page = b"<html><script>steal_cookies()</script></html>"
print(upload_v2("neha.png", png))
print(upload_v2("profile.html", page))
print(upload_v2("photo.png", page))
print(upload_v2("index.html.png", page))
print(upload_v2("big.pdf", b"%PDF-" + b"0" * 3_000_000))

Real output (the random name differs each run):

saved neha.png as adffe90c09e800a7.png
rejected profile.html: extension .html not allowed
rejected photo.png: content is not really .png
rejected index.html.png: content is not really .png
rejected big.pdf: too big

Problem 1c: storing and scanning uploads

  • Store outside the webroot: OWASP, "Store the files on a different server. If that's not possible, store them outside of the webroot." A file then cannot be opened directly by its address; your code serves it, after checking who may see it.
  • Malware scanning: malware (episode 5) is software built to do harm. OWASP: "Run the file through an antivirus or a sandbox if available". Antivirus compares files with signatures of known malware; a sandbox is a sealed test machine where a file is opened and watched.
  • Only authorised users upload: "Only allow authorized users to upload files".
  • Defence in depth (episode 3): every single check can be fooled, so stack them.

Problem 2: path traversal

  • A path is a file's address on disk, such as invoices/inv-501.pdf.
  • In a path, .. means the folder one level up, and . means "this folder".
  • An absolute path starts at the top of the disk (/etc/hostname); a relative path starts from some folder (invoices/x).
  • Path traversal (also directory traversal) means using .., absolute paths or similar tricks in a file name to reach files outside the folder the app meant to use.

Real-life story: "Download your invoice" uses ?file=inv-501.pdf. Someone sends ../config/db.txt instead.

Demo 3: path traversal on a toy folder

# Download an invoice by name: /download?file=inv-501.pdf
import os, tempfile
root = os.path.realpath(tempfile.mkdtemp())
BASE = os.path.join(root, "invoices")
os.makedirs(BASE); os.makedirs(os.path.join(root, "config"))
open(os.path.join(BASE, "inv-501.pdf"), "w").write("Invoice 501: Rs 1,299")
open(os.path.join(root, "config", "db.txt"), "w").write("DB_PASSWORD=toy-pass-123")

def download_v1(name):
    return open(os.path.join(BASE, name)).read()

def download_v2(name):
    path = os.path.realpath(os.path.join(BASE, name))   # canonical form
    if os.path.commonpath([path, BASE]) != BASE:
        return "403 Forbidden: outside the invoices folder"
    return open(path).read()

for name in ["inv-501.pdf", "../config/db.txt"]:
    print(f"v1 {name!r:20} -> {download_v1(name)}")
for name in ["inv-501.pdf", "../config/db.txt", "/etc/hostname"]:
    print(f"v2 {name!r:20} -> {download_v2(name)}")

Real output:

v1 'inv-501.pdf'        -> Invoice 501: Rs 1,299
v1 '../config/db.txt'   -> DB_PASSWORD=toy-pass-123
v2 'inv-501.pdf'        -> Invoice 501: Rs 1,299
v2 '../config/db.txt'   -> 403 Forbidden: outside the invoices folder
v2 '/etc/hostname'      -> 403 Forbidden: outside the invoices folder
  • os.path.realpath gives the canonical form of a path: absolute, with every ., .. and symbolic link worked out.
  • os.path.commonpath([a, b]) gives the longest shared folder of two paths, comparing whole folder names.
  • Watch out: os.path.join("/srv/invoices", "/etc/passwd") returns /etc/passwd. If the second part is absolute, join throws the first part away. Checking the final resolved path catches this too.

Problem 2b: canonicalisation

Real-life example: one flat, many addresses ("Flat 4B", "2nd floor, left", "opposite the lift"). A guard first writes the one official address, then checks it.

  • URL encoding writes characters as % plus two hex digits: / is %2f, . is %2e. Servers decode it before using the value.
  • Normalising a path means rewriting it in its simplest form (a/x/../b becomes a/b).
  • Canonicalisation means turning every possible spelling of a value into one standard (canonical) form, and only then checking it.

Demo 4: why "just block ../" fails

# Why "just block ../" fails: check the FINAL form, not the raw text
from urllib.parse import unquote
import posixpath

def filter_v1(raw):                      # check the raw text, decode later
    if "../" in raw:
        return "blocked"
    return "allowed -> " + unquote(raw)

def filter_v2(raw):                      # delete "../" once
    return "allowed -> " + raw.replace("../", "")

def canonical(raw):                      # decode, normalise, THEN check
    path = posixpath.normpath("/invoices/" + unquote(raw))
    if not path.startswith("/invoices/"):
        return "blocked -> " + path
    return "allowed -> " + path

print("v1 ..%2fconfig/db.txt  :", filter_v1("..%2fconfig/db.txt"))
print("v2 ....//config/db.txt :", filter_v2("....//config/db.txt"))
for raw in ["inv-501.pdf", "..%2fconfig/db.txt", "../../config/db.txt"]:
    print(f"canonical {raw:20}:", canonical(raw))

Real output:

v1 ..%2fconfig/db.txt  : allowed -> ../config/db.txt
v2 ....//config/db.txt : allowed -> ../config/db.txt
canonical inv-501.pdf         : allowed -> /invoices/inv-501.pdf
canonical ..%2fconfig/db.txt  : blocked -> /config/db.txt
canonical ../../config/db.txt : blocked -> /config/db.txt

v1 checked the raw text before decoding; v2 deleted ../ once, which turns ....// into ../. Never try to "clean" a dangerous path. Decode, normalise, resolve, then check against the allowed folder. Best of all, don't use user-supplied names: look files up by an ID stored in your database (see the mini project).

Problem 3: zip slip

  • An archive is one file that holds many files, each with its own stored name: .zip, .tar, .jar.
  • Extracting (unpacking) writes each entry to disk under its stored name.
  • Zip slip: an archive entry whose name contains ../ (or an absolute path) is written outside the destination folder. Snyk named it and disclosed it publicly on 5 June 2018; their research calls it "a form of directory traversal" exploited through archive extraction, affecting "thousands of projects" (including ones from HP, Amazon, Apache and Pivotal) across JavaScript, Ruby, .NET, Go and especially Java.

Demo 5: zip slip on a toy archive

# Zip slip: an uploaded zip whose entry name climbs out of the folder
import zipfile, io, os, tempfile
root = os.path.realpath(tempfile.mkdtemp())
DEST = os.path.join(root, "uploads", "batch1"); os.makedirs(DEST)

buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as z:      # a toy archive we build ourselves
    z.writestr("photo1.png", "pixels")
    z.writestr("../../app_config.py", "DEBUG = True  # overwritten!")

def extract_v1(zf):                            # trusts every entry name
    for name in zf.namelist():
        out = os.path.join(DEST, name)
        open(out, "wb").write(zf.read(name))

def extract_v2(zf):                            # canonicalise, then check
    for name in zf.namelist():
        out = os.path.realpath(os.path.join(DEST, name))
        if os.path.commonpath([out, DEST]) != DEST:
            print("  skipped unsafe entry:", name); continue
        open(out, "wb").write(zf.read(name))

extract_v1(zipfile.ZipFile(buf))
print("v1 wrote outside uploads:", os.path.exists(f"{root}/app_config.py"))
os.remove(f"{root}/app_config.py")
extract_v2(zipfile.ZipFile(buf))
print("v2 wrote outside uploads:", os.path.exists(f"{root}/app_config.py"))

Real output:

v1 wrote outside uploads: True
  skipped unsafe entry: ../../app_config.py
v2 wrote outside uploads: False

Library safety nets (both checked in Python 3.13 on 12 Nov 2026):

  • zipfile.ZipFile.extractall() already removes .. from entry names, so the same archive's bad entry lands inside the destination as app_config.py.
  • tarfile takes filter="data", which refuses entries that would land outside the destination:
import tarfile, io, os, tempfile
root = os.path.realpath(tempfile.mkdtemp()); dest = os.path.join(root, "out"); os.makedirs(dest)
buf = io.BytesIO()
with tarfile.open(fileobj=buf, mode="w") as t:
    data = b"DEBUG = True"
    info = tarfile.TarInfo("../app_config.py"); info.size = len(data)
    t.addfile(info, io.BytesIO(data))
buf.seek(0)
try:
    tarfile.open(fileobj=buf).extractall(dest, filter="data")
except tarfile.FilterError as e:
    print(type(e).__name__ + ":", e)

Real output (the temporary path differs each run):

OutsideDestinationError: '../app_config.py' would be extracted to '/tmp/tmppxkdogct/app_config.py', which is outside the destination

Also limit the number and total size of entries: a small compressed archive can unpack to gigabytes (a "zip bomb"). Practice 2 below adds those limits.

Problem 4: serialisation and insecure deserialisation

  • Serialisation turns an object in memory into bytes or text so it can be saved (file, cookie, database) or sent (network, message queue).
  • Deserialisation turns those bytes back into a live object.
  • pickle is Python's built-in serialisation module. Java serialization uses the Serializable interface with ObjectOutputStream / ObjectInputStream.
  • Insecure deserialisation means deserialising bytes an attacker could have written or changed, with a format that can do more than hold data.
  • Base64 (episode 10) is an encoding, not encryption: anyone can decode and re-encode it.
  • A cookie is a small value the server asks the browser to keep and send back; the user can read and change it.

Real-life example: flat-pack furniture. The box holds parts and instructions, and you follow every step. A pickle is the same: data plus steps to rebuild it, and one step can be "call this function". That step is an object's __reduce__ method, which returns a callable and its arguments.

The Python documentation's own warning: "The pickle module is not secure. Only unpickle data you trust." and "It is possible to construct malicious pickle data which will execute arbitrary code during unpickling."

This is the same harmless demonstration the Python docs use: the "payload" only calls print.

# The shop saves a shopping cart in a cookie with pickle
import pickle, base64

cart = {"user": "neha", "items": ["charger", "cable"]}
cookie = base64.b64encode(pickle.dumps(cart))
print("cookie:", cookie[:40], "...")
print("loaded:", pickle.loads(base64.b64decode(cookie)))

# But the cookie comes back from the BROWSER. Anyone can edit it.
class NotACart:
    def __reduce__(self):            # "to rebuild me, call this function"
        return (print, ("!!! code ran while loading the cookie !!!",))

evil_cookie = base64.b64encode(pickle.dumps(NotACart()))
print("loading the edited cookie...")
pickle.loads(base64.b64decode(evil_cookie))      # the server just ran print()

class SafeUnpickler(pickle.Unpickler):          # allow no functions at all
    def find_class(self, module, name):
        raise pickle.UnpicklingError(f"global '{module}.{name}' is forbidden")

import io
try:
    SafeUnpickler(io.BytesIO(base64.b64decode(evil_cookie))).load()
except pickle.UnpicklingError as e:
    print("SafeUnpickler:", e)

Real output (the cookie text is the same each run):

cookie: b'gASVMQAAAAAAAAB9lCiMBHVzZXKUjARuZWhhlIwF' ...
loaded: {'user': 'neha', 'items': ['charger', 'cable']}
loading the edited cookie...
!!! code ran while loading the cookie !!!
SafeUnpickler: global 'builtins.print' is forbidden

find_class is the method the unpickler calls whenever the data asks for a class or function; overriding it (the docs' "Restricting Globals" section) lets you forbid all of them or allow a small safe set. It reduces the risk, but the real rule is: never unpickle anything a user can touch.

Problem 4b: gadget chains (the idea)

  • A gadget is an ordinary class that already exists in your app or its libraries, whose code runs automatically while an object is being rebuilt (for example a special read method, a setter or a hash function).
  • A gadget chain arranges serialised objects so that rebuilding A calls B, which calls C, which finally does something harmful. The attacker sends no new code, only an arrangement of your own classes, which is why patching one class never ends the problem.
  • The defence does not depend on knowing the chains: don't deserialise untrusted data; if you must, allow only the exact classes you expect.

Demo 7: Java serialization with an allowlist filter

import java.io.*;
import java.util.*;

public class Main {
    record Cart(String user, int items) implements Serializable {}

    static byte[] save(Object o) throws IOException {
        var out = new ByteArrayOutputStream();
        new ObjectOutputStream(out).writeObject(o);
        return out.toByteArray();
    }
    static Object load(byte[] bytes, String filter) throws Exception {
        var in = new ObjectInputStream(new ByteArrayInputStream(bytes));
        if (filter != null)   // allowlist: only these classes may be created
            in.setObjectInputFilter(
                ObjectInputFilter.Config.createFilter(filter));
        return in.readObject();
    }
    public static void main(String[] args) throws Exception {
        String allow = "Main$Cart;java.lang.*;!*";
        System.out.println(load(save(new Cart("neha", 2)), allow));
        byte[] other = save(new HashMap<>(Map.of("is_admin", true)));
        System.out.println("no filter: " + load(other, null));
        try { load(other, allow); }
        catch (InvalidClassException e) {
            System.out.println("filter: " + e.getMessage()); }
    }
}

Real output:

Cart[user=neha, items=2]
no filter: {is_admin=true}
filter: filter status: REJECTED
  • ObjectInputFilter came with JEP 290, "Filter Incoming Serialization Data", in Java 9. A pattern like Main$Cart;java.lang.*;!* is read left to right: ; separates rules, a trailing * matches a prefix, and ! rejects. So: allow Cart, allow java.lang classes, reject everything else.
  • Oracle's Java 21 docs: "A serialization filter is not enabled or configured by default." You must switch one on (per stream as here, or JVM-wide with the jdk.serialFilter property). An allowlist is safer than a list of known-bad classes.

Problem 4c: the real fix, plain data with a schema

  • JSON (JavaScript Object Notation) can hold only objects (dictionaries), arrays (lists), strings, numbers, true, false and null. It cannot name a class or a function, so loading it never runs code. The pickle docs: "Safer serialization formats such as json may be more appropriate if you are processing untrusted data."
  • A schema is a written rule for the exact shape of the data: which keys, which types, which limits. Validating against it refuses anything unplanned.

Demo 8: JSON with a schema check

# The fix: plain data (JSON) + a schema check. JSON can't call functions.
import json

SCHEMA = {"user": str, "items": list}           # exactly these keys and types

def load_cart(text):
    data = json.loads(text)             # only dicts, lists, text, numbers
    if not isinstance(data, dict) or set(data) != set(SCHEMA):
        raise ValueError(f"wrong keys: {sorted(data)}")
    for key, typ in SCHEMA.items():
        if not isinstance(data[key], typ):
            raise ValueError(f"{key} must be a {typ.__name__}")
    items = data["items"]
    if len(items) > 50 or not all(isinstance(i, str) for i in items):
        raise ValueError("items: at most 50 strings")
    return data

tests = ['{"user": "neha", "items": ["charger", "cable"]}',
         '{"user": "neha", "items": ["charger"], "is_admin": true}',
         '{"user": "neha", "items": "charger"}',
         '{"__reduce__": "print"}']
for t in tests:
    try:
        print("OK     ", load_cart(t))
    except ValueError as e:
        print("REFUSED", e)

Real output:

OK      {'user': 'neha', 'items': ['charger', 'cable']}
REFUSED wrong keys: ['is_admin', 'items', 'user']
REFUSED items must be a list
REFUSED wrong keys: ['__reduce__']

If the user must not change the data at all, either keep it on the server and put only a random ID in the cookie, or sign it with an HMAC (episode 12) and verify the signature before using it.

The whole chain

  1. A web page was saved as a "photo" → extension allowlist, magic bytes, size limit.
  2. The user's name went into the web folder → our own random name, storage outside the webroot, malware scanning.
  3. ../ read the password file → resolve with realpath, check with commonpath.
  4. Simple filters were bypassed → canonicalise first, or use IDs instead of names.
  5. A zip wrote outside its folder → check every entry, use safe extraction, limit size.
  6. A cookie ran code → no pickle on user data; JSON with a schema; Java filters as a backstop.

Common mistakes (and what they cause)

Mistake Real message / result Why it's wrong Fix
Trusting Content-Type saved profile.html in the web folder the label is user-chosen extension allowlist + magic bytes
Saving under the user's file name web folder now: ['index.html', ...] (home page replaced) the user chooses where it goes random name, outside the webroot
open(os.path.join(BASE, name)) v1 '../config/db.txt' -> DB_PASSWORD=toy-pass-123 .. and absolute paths escape realpath + commonpath
Check, then decode v1 ..%2fconfig/db.txt : allowed -> ../config/db.txt checked a spelling, not the meaning decode, normalise, then check
path.startswith(BASE) ../files_backup/x broken=True compares text, not folders os.path.commonpath([path, BASE]) == BASE
Extracting archive names as-is v1 wrote outside uploads: True entry names are user input per-entry check, filter="data" for tar
pickle.loads(cookie) !!! code ran while loading the cookie !!! pickle is instructions JSON + schema, or server-side storage
JSON with single quotes json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) JSON needs double quotes send real JSON (json.dumps)
Unfiltered ObjectInputStream no filter: {is_admin=true} any class on the class path loads ObjectInputFilter allowlist

Mini project: a safe document store

Combines everything: allowlist + magic bytes + size limit, random IDs, storage outside any webroot, downloads by ID only with an owner check (the same 404 for "not yours" and "doesn't exist"), and metadata stored as JSON.

# Mini project: a safe document store for the shop
# - uploads: allowlist + magic bytes + size limit + random id, stored outside the webroot
# - downloads: by id only (no user file names), and only by the owner
# - metadata saved as JSON (plain data), never pickle
import json, os, secrets, tempfile

class DocStore:
    TYPES = {".pdf": b"%PDF-", ".png": b"\x89PNG\r\n\x1a\n", ".jpg": b"\xff\xd8\xff"}
    MAX = 2_000_000

    def __init__(self):
        self.dir = os.path.realpath(tempfile.mkdtemp())   # not under any webroot
        self.meta_path = os.path.join(self.dir, "meta.json")
        self.meta = {}

    def upload(self, user, filename, data):
        ext = os.path.splitext(filename)[1].lower()
        if ext not in self.TYPES:
            return None, "extension not allowed"
        if not data.startswith(self.TYPES[ext]):
            return None, "content does not match extension"
        if len(data) > self.MAX:
            return None, "file too big"
        doc_id = secrets.token_hex(12)
        with open(os.path.join(self.dir, doc_id + ext), "wb") as f:
            f.write(data)
        self.meta[doc_id] = {"owner": user, "ext": ext, "original": filename[:100]}
        with open(self.meta_path, "w") as f:
            json.dump(self.meta, f)                        # plain data on disk
        return doc_id, "saved"

    def download(self, user, doc_id):
        entry = self.meta.get(doc_id)
        if entry is None or entry["owner"] != user:      # same answer for both
            return None, "404 not found"
        with open(os.path.join(self.dir, doc_id + entry["ext"]), "rb") as f:
            return f.read(), "200 ok"

store = DocStore()
doc, msg = store.upload("neha", "invoice.pdf", b"%PDF-1.7 toy invoice")
print("neha upload   :", msg, "| id length", len(doc))
print("bad upload    :", store.upload("neha", "evil.html", b"<html>")[1])
print("fake pdf      :", store.upload("neha", "x.pdf", b"<html>")[1])
print("neha download :", store.download("neha", doc))
print("ravi download :", store.download("ravi", doc)[1])
print("path in id    :", store.download("neha", "../meta")[1])

Real output:

neha upload   : saved | id length 24
bad upload    : extension not allowed
fake pdf      : content does not match extension
neha download : (b'%PDF-1.7 toy invoice', '200 ok')
ravi download : 404 not found
path in id    : 404 not found

Practice

  1. Add GIF images (GIF87a or GIF89a) to the upload allowlist.
  2. Make the zip extractor refuse archives with more than 100 entries or more than 10 MB unpacked, without trusting the sizes the archive declares.
  3. Break the quiz's startswith check with a real input, then fix it.
  4. List three places in apps you use where data is deserialised (cookies, caches, message queues, saved files) and say which format you would use.
Practice 1 solution
# Practice 1 solution: allow GIF images (two possible magic headers)
import os
ALLOWED = {".png": (b"\x89PNG\r\n\x1a\n",), ".jpg": (b"\xff\xd8\xff",),
           ".pdf": (b"%PDF-",), ".gif": (b"GIF87a", b"GIF89a")}

def check(filename, data):
    ext = os.path.splitext(filename)[1].lower()
    if ext not in ALLOWED:
        return f"rejected {filename}: extension not allowed"
    if not data.startswith(ALLOWED[ext]):          # startswith accepts a tuple
        return f"rejected {filename}: content is not really {ext}"
    return f"accepted {filename}"

print(check("cat.gif", b"GIF89a" + b"\x01\x00..."))
print(check("old.gif", b"GIF87a" + b"\x01\x00..."))
print(check("fake.gif", b"<html>hi</html>"))
accepted cat.gif
accepted old.gif
rejected fake.gif: content is not really .gif
Practice 2 solution
# Practice 2 solution: safe extraction with limits on entries and total size
import zipfile, io, os, tempfile
root = os.path.realpath(tempfile.mkdtemp())
DEST = os.path.join(root, "uploads"); os.makedirs(DEST)
MAX_ENTRIES, MAX_TOTAL = 100, 10_000_000           # 100 files, 10 MB unpacked

def extract_safe(zf):
    infos = zf.infolist()
    if len(infos) > MAX_ENTRIES:
        return f"refused: {len(infos)} entries (max {MAX_ENTRIES})"
    total = sum(i.file_size for i in infos)        # sizes declared in the zip
    if total > MAX_TOTAL:
        return f"refused: {total:,} bytes unpacked (max {MAX_TOTAL:,})"
    written = 0
    for info in infos:
        out = os.path.realpath(os.path.join(DEST, info.filename))
        if os.path.commonpath([out, DEST]) != DEST or info.is_dir():
            continue
        data = zf.read(info)
        written += len(data)
        if written > MAX_TOTAL:                    # don't trust declared sizes
            return "refused: real size over the limit"
        open(out, "wb").write(data)
    return f"extracted {len(infos)} entries, {written:,} bytes"

def make_zip(entries):
    buf = io.BytesIO()
    with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as z:
        for name, data in entries: z.writestr(name, data)
    return zipfile.ZipFile(buf)

print(extract_safe(make_zip([("a.png", b"pixels"), ("b.png", b"more")])))
print(extract_safe(make_zip([(f"f{i}.txt", b"x") for i in range(500)])))
print(extract_safe(make_zip([("huge.txt", b"0" * 50_000_000)])))
extracted 2 entries, 10 bytes
refused: 500 entries (max 100)
refused: 50,000,000 bytes unpacked (max 10,000,000)
Practice 3 solution
# Practice 3 solution: the quiz check, broken and fixed
import os
BASE = "/srv/files"

def safe_broken(name):
    path = os.path.realpath(os.path.join(BASE, name))
    return path.startswith(BASE)                   # compares TEXT

def safe_fixed(name):
    path = os.path.realpath(os.path.join(BASE, name))
    return os.path.commonpath([path, BASE]) == BASE   # compares folders

for name in ["report.pdf", "../files_backup/x", "../../etc/passwd"]:
    print(f"{name:20} broken={safe_broken(name)!s:5}  fixed={safe_fixed(name)}")
report.pdf           broken=True   fixed=True
../files_backup/x    broken=True   fixed=False
../../etc/passwd     broken=False  fixed=False
Practice 4 discussion

Typical places: session or cart cookies, server caches (Redis/memcached), background job queues, files saved by desktop apps, machine-learning model files. For anything a user or another system could change, use a plain data format (JSON, or Protocol Buffers with a fixed schema), validate it against a schema, and sign it (HMAC) if it must not be changed. Never use pickle or unfiltered Java serialization across a trust boundary.

Quick quiz

BASE = "/srv/files"

def safe(name):
    path = os.path.realpath(os.path.join(BASE, name))
    return path.startswith(BASE)

Is this path check safe? A: yes, realpath fixes everything. B: no, "../files_backup/x" passes. C: no, realpath is too slow. D: yes, but only on Windows.

Answer

B. /srv/files_backup/x starts with the text /srv/files, so startswith returns True even though it is a different folder. Compare whole folder names: os.path.commonpath([path, BASE]) == BASE, or path.startswith(BASE + os.sep). Practice 3 shows it running.

Interview questions

  1. Why isn't checking the Content-Type enough for uploads? It is a label the client writes and can be spoofed. Check the extension against an allowlist and the magic bytes, limit size, rename, store outside the webroot and scan.
  2. Why rename uploaded files? So the user can't overwrite files, inject odd characters, or traverse folders through the name; the server controls the name.
  3. What is path traversal and how do you prevent it? Using .. or absolute paths in a file name to escape a folder. Avoid user-supplied names (use IDs); otherwise canonicalise (realpath) and check the result is inside the base folder by comparing path components.
  4. What is canonicalisation, and why does order matter? Converting a value to one standard form before checking it. Checking before decoding or normalising lets other spellings (%2f, ....//) through.
  5. What is zip slip? Path traversal through archive entry names during extraction; fix with per-entry canonical checks or library safe modes, plus size and count limits.
  6. Why is pickle.loads on untrusted data dangerous? Pickle data can tell the loader to call any function (__reduce__), so loading it can execute arbitrary code.
  7. What is a gadget chain? A sequence of existing classes whose automatic rebuild code calls each other and ends in a harmful action; the attacker supplies only serialised data.
  8. How do you deserialise safely in Java? Avoid native serialization for untrusted input; if unavoidable, use an ObjectInputFilter allowlist (JEP 290), and limit depth and size.
  9. What should replace native serialization? A data-only format such as JSON with schema validation, and a signature if integrity matters.

Glossary

Term Meaning, with an example
Upload Sending a file from your device to a server, e.g. a profile photo.
Unrestricted file upload Accepting files without checking type, name, size or destination; e.g. a "photo" that is an HTML page.
Content-Type / MIME type A label in the request saying what the data claims to be, e.g. image/png; chosen by the client, can be spoofed.
Spoofing Faking a value to look legitimate, e.g. writing image/png on a web page.
Extension The end of a file name after the last dot, e.g. .pdf; user-chosen.
Allowlist A list of the only accepted values, e.g. .png .jpg .pdf; the opposite of a blocklist.
Magic bytes / file signature The first bytes that mark a file's real format, e.g. %PDF- for PDF.
Webroot The folder a web server serves to the world, e.g. /var/www/html.
Storing outside the webroot Keeping uploads where no URL reaches them, served only through code that checks access.
Renaming uploads Replacing the user's file name with one the server generates, e.g. 8d99de4877cdf0b2.png.
Malware scanning Checking uploads with antivirus signatures or a sandbox before anyone opens them.
Antivirus Software that recognises known malware by signatures and behaviour.
Sandbox A sealed test environment where a file is opened and watched safely.
Path A file's address on disk, e.g. invoices/inv-501.pdf.
.. "The folder one level up" in a path.
Absolute / relative path A path from the top of the disk (/etc/hostname) vs from some folder (config/db.txt).
Path (directory) traversal Using .. or absolute paths to reach files outside the intended folder.
Canonical form The one standard, fully resolved form of a value, e.g. realpath("/a/x/../b") → /a/b.
Canonicalisation Converting every spelling to the canonical form before checking it.
URL encoding Writing characters as % + hex, e.g. / as %2f.
Normalising Rewriting a path in its simplest form, e.g. a/./b → a/b.
realpath / commonpath Python functions that resolve a path completely / find the shared folder of paths.
Archive One file holding many named files, e.g. .zip, .tar, .jar.
Zip slip Path traversal through archive entry names during extraction (Snyk, 2018).
Zip bomb A small archive that unpacks to a huge size.
Serialisation Turning an object into bytes/text to save or send, e.g. pickle.dumps(cart).
Deserialisation Turning those bytes back into an object, e.g. pickle.loads(data).
Insecure deserialisation Deserialising data an attacker could control with a format that can do more than hold data.
pickle Python's built-in serialisation module; "not secure" for untrusted data.
__reduce__ The method telling pickle how to rebuild an object: a function plus its arguments.
find_class The unpickler method called for every class/function the data asks for; override it to restrict them.
Gadget An existing class whose code runs automatically during deserialisation.
Gadget chain Gadgets arranged so rebuilding one triggers the next, ending in a harmful action.
Java serialization Java's built-in object-to-bytes mechanism (Serializable, ObjectInputStream).
ObjectInputFilter Java's filter (JEP 290, Java 9) that accepts or rejects classes during deserialisation.
Base64 An encoding of bytes as text; not encryption, anyone can reverse it.
Cookie A small value the browser stores and sends back; the user can read and edit it.
JSON A data-only text format: objects, arrays, strings, numbers, true, false, null.
Schema A written rule for the exact keys, types and limits data must have.
HMAC A keyed signature that proves data was not changed (episode 12).
Defence in depth Stacking several independent checks so one failure isn't fatal (episode 3).

Summary

  • A file's name and Content-Type are user input. Check the extension against an allowlist, the magic bytes and the size; give the file your own random name; store it outside the webroot; scan it; let only authorised users upload.
  • Path traversal: never glue a user's name onto a folder. Use IDs, or resolve with realpath and check with commonpath.
  • Canonicalise before checking: decode and normalise first; never "clean" a path.
  • Zip slip: archive entry names are user input; check each one, use filter="data" for tar, limit count and size.
  • Insecure deserialisation: pickle and native Java serialization can run code while loading. Use JSON with a schema; sign data you hand to users; use ObjectInputFilter allowlists where Java serialization can't be avoided.

Next episode: authentication failures and security logging failures: brute force, throttling vs lockout, CAPTCHA, account enumeration, password resets, remember-me tokens, and logging and alerting.