SyntaxPilot
Courses/System Design/Lesson 30
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

System Design · Lesson 30 of 40

Design the Instagram feed

queued

These are the notes for System Design episode 30, our fourth full design problem. Priya opens Instagram on the metro at 8 pm and a home feed of posts from the 500 accounts she follows appears almost instantly. Today you design that feed: you'll write the requirements, build the feed the obvious way at read time and watch it take 500 queries and about a second per open, flip it to fan-out on write so opening is one read, break that with a star with 100 million followers, fix it with the hybrid feed (push for most accounts, pull for stars), learn why Instagram stopped showing posts newest first (people missed on average 70 percent of their feeds), rank posts with a score, stop repeated posts while scrolling with cursor pagination, and put the photos behind object storage and a CDN. Every demo is plain Python and runs as-is; outputs are real (Python 3.13). The sizes (500 follows, 2 ms per query, 100 million followers, a million writes a second) are our own assumptions for the exercise; star_kabir is a made-up account.

A feed is a list of posts chosen for one user, newest or most relevant first, that they scroll through. Example: the Instagram home feed, the X/Twitter timeline, the LinkedIn feed.

The problems

Problem What it teaches
Priya opens Instagram Requirements; reads far outnumber writes
Asking 500 accounts on every open Fan-out on read and why it's slow at read time
Build the feed in advance Fan-out on write: a ready-made feed per user
A star with 100 million followers The celebrity problem; the hybrid feed (push + pull, merged)
Neha's news, buried Ranking: candidates, a score, interest, closeness, recency
The same posts twice while scrolling Offset vs cursor pagination
Millions of photos Object storage + CDN; the feed holds only ids
Put it all together Post service, queue, fan-out workers, feed cache, feed service

Problem 0: Priya opens Instagram (requirements)

Functional requirements:

  • Show posts from the accounts you follow.
  • A new post reaches followers within seconds.
  • Infinite scroll: keep loading older posts, with no repeats and no gaps.

Non-functional requirements:

  • Opening the feed is fast: well under a second.
  • Reads >> writes: people scroll far more often than they post, so reading must be the cheap path.
  • Some accounts have 100 million followers.

A few words first:

  • To follow an account means asking to see its posts in your feed. The people who follow you are your followers; the accounts you follow are your followees.
  • A post is one item someone shares: a photo or video with a caption, an author and a time.
  • A query is one question sent to a database. Example: "the latest 3 posts by chai_point".

Problem 1: asking 500 accounts on every open (fan-out on read vs write)

Real life: the morning newspaper

Getting the news Building a feed
Call 500 friends every morning: "any news?" Fan-out on read: when you open the app, query every account you follow
The paper is already at your door, printed at night Fan-out on write: when someone posts, put it into every follower's ready-made feed

Fan-out means one thing spreading out to many. On read means the spreading happens when someone reads (opens the feed); on write means it happens when someone writes (posts).

The obvious way: fan-out on read

import time

following = [f"user{i}" for i in range(500)]   # Priya follows 500 accounts
posts = {u: [(i * 7, f"{u}-post{i}") for i in range(3)] for u in following}

def latest_posts(user):
    time.sleep(0.002)                     # one database query: ~2 ms
    return posts[user]

def open_feed():                          # built when she opens the app
    all_posts = []
    for u in following:                   # ask every account she follows
        all_posts += latest_posts(u)
    all_posts.sort(reverse=True)          # newest first
    return all_posts[:20]

start = time.perf_counter()
feed = open_feed()
took = time.perf_counter() - start
print(f"queries: {len(following)}  feed built in {took:.1f} s")
print(f"10 million opens a minute: {10_000_000 * 500 / 60:,.0f} queries/s")

Output:

queries: 500  feed built in 1.1 s
10 million opens a minute: 83,333,333 queries/s
  • time.sleep(0.002) stands in for one database query of about 2 ms.
  • One open = 500 queries, merged and sorted, about a second (it varies a little from run to run). And it happens every time anyone opens the app.
  • Ten million opens a minute is 83 million queries a second just to show feeds.

The work happens at read time, but reads are the common case. That's the wrong way round.

The fix: fan-out on write

Every user gets a ready-made feed: a list of post ids, newest first, capped (here at 500), kept in a fast store like Redis (episode 7). When someone posts, we push the post id to the front of each follower's list. Opening the app is just reading your own list.

from collections import defaultdict, deque

followers = defaultdict(list)             # author -> who follows them
feeds = defaultdict(lambda: deque(maxlen=500))   # user -> ready-made feed
writes = 0

def follow(user, author):
    followers[author].append(user)

def post(author, post_id):                # fan-out ON WRITE
    global writes
    for f in followers[author]:
        feeds[f].appendleft(post_id)      # push into each follower's feed
        writes += 1

for u in ["priya", "rahul", "neha"]:
    follow(u, "chai_point")
follow("priya", "amit")
post("chai_point", "p1: masala chai offer")
post("amit", "p2: Goa photos")
print("priya opens app ->", list(feeds["priya"])[:20], "(1 read)")
print("writes so far:", writes)

celebrity = 100_000_000                   # followers of a film star
per_second = 1_000_000                    # feed writes the system can do
wait = celebrity / per_second
print(f"star posts -> {celebrity:,} writes, last fan waits {wait:.0f} s")

Output:

priya opens app -> ['p2: Goa photos', 'p1: masala chai offer'] (1 read)
writes so far: 4
star posts -> 100,000,000 writes, last fan waits 100 s
  • defaultdict(list) is a dictionary that creates an empty list the first time you use a new key.
  • deque(maxlen=500) is a list that throws away the oldest items once it holds 500; appendleft adds to the front.
  • Two posts cost 4 writes (one per follower) but Priya's open is 1 read.
  • The last line is the catch: a star with 100 million followers posts, that's 100 million writes; at a million writes a second the last fan waits 100 seconds, and the system is flooded every time any star posts.

Problem 2: a star with 100 million followers (the hybrid feed)

Real life: letters vs a notice board

Sharing news The hybrid feed
A friend writes you a letter: it lands in your box Normal accounts: push (fan-out on write)
A star puts one notice on a board; fans go and look Stars: pull (fan-out on read, only for them)
You read your letters and the board together Merge by time at read time

The celebrity problem (also called the hot-key problem) is when a few accounts have so many followers that fan-out on write becomes enormous for each of their posts.

import heapq

STAR_LIMIT = 1_000_000                # more followers = a "star"
follower_count = {"chai_point": 2_400, "amit": 310,
                  "star_kabir": 48_000_000}

# pushed at write time into Priya's ready feed: (time, post)
ready_feed = [(1005, "amit: Goa photos"),
              (990, "chai_point: masala chai offer")]
# stars' posts are NOT pushed; they sit in the star's own list
star_posts = {"star_kabir": [(1010, "star_kabir: new film trailer"),
                            (900, "star_kabir: gym selfie")]}
priya_follows = ["chai_point", "amit", "star_kabir"]

def open_feed():
    stars = [a for a in priya_follows if follower_count[a] > STAR_LIMIT]
    lists = [ready_feed] + [star_posts[s] for s in stars]  # pull now
    merged = heapq.merge(*lists, reverse=True)       # newest first
    return [p for _, p in merged], stars

feed, stars = open_feed()
print("stars pulled at read time:", stars)
for p in feed:
    print("  ", p)

Output:

stars pulled at read time: ['star_kabir']
   star_kabir: new film trailer
   amit: Goa photos
   chai_point: masala chai offer
   star_kabir: gym selfie
  • Any account with more than STAR_LIMIT followers counts as a star. Its posts are not pushed; they sit in the star's own list.
  • open_feed finds the stars Priya follows, pulls their lists now, and merges them with her ready feed.
  • heapq.merge(..., reverse=True) merges lists that are each already sorted (newest first) into one sorted list, without re-sorting everything.
  • The result is in time order: the trailer (1010), Goa photos (1005), the chai offer (990), the older selfie (900).

A star's post is now one write, not 100 million. Each open pulls only a handful of star lists, and those lists are read so often that they stay in cache. That's the hybrid feed: fan-out on write for most accounts, fan-out on read for stars.

Problem 3: Neha's news, buried (ranking)

Neha, Priya's sister, posted "Got the job!" six hours ago. Since then 200 newer posts arrived (memes, news, offers). Newest first, Neha's post sinks; Priya stops scrolling before she reaches it.

Instagram described exactly this in March 2016: "people miss on average 70 percent of their feeds." It announced that "The order of photos and videos in your feed will be based on the likelihood you'll be interested in the content", together with your relationship with the person posting and how recent the post is.

Ranking means ordering items by a score that estimates how much this user will care, instead of by time alone.

posts = [  # (author, hours ago, how close Priya is to them, likes per view)
    ("random_meme_page", 0.5, 0.1, 0.02),
    ("college_friend",   1.0, 0.9, 0.05),
    ("chai_point",       2.0, 0.3, 0.10),
    ("sister_neha",      6.0, 1.0, 0.30),
    ("news_channel",     0.2, 0.2, 0.01),
]

def score(p):
    author, hours, closeness, interest = p
    return (closeness * 2 + interest * 10) / (hours + 2) ** 1.5  # old=low

print("newest first:", [p[0] for p in sorted(posts, key=lambda p: p[1])])
print("ranked:      ", [p[0] for p in sorted(posts, key=score, reverse=True)])

Output:

newest first: ['news_channel', 'random_meme_page', 'college_friend', 'chai_point', 'sister_neha']
ranked:       ['college_friend', 'sister_neha', 'chai_point', 'news_channel', 'random_meme_page']
  • The three signals: closeness (how close Priya is to the author: family, friends you message), interest (here, likes per view: how much people engage with it) and recency (age in hours).
  • score adds closeness and interest, then divides by (hours + 2) ** 1.5, so older posts slowly sink. The + 2 stops brand-new posts from getting a near-infinite boost.
  • Newest first puts sister_neha last; ranked, she's second, and random_meme_page drops to the bottom.

Real systems don't hand-write this formula: they learn it with machine learning from what people actually like, comment on and spend time on. The shape stays the same: fetch a few hundred candidate posts (the hybrid feed), score each one, sort.

Problem 4: the same posts twice while scrolling (cursor pagination)

Pagination means loading a long list one page at a time. Infinite scroll asks for the next page as you near the bottom.

feed = [f"post{i}" for i in range(10, 0, -1)]   # newest first

def page_by_offset(offset, size=3):
    return feed[offset:offset + size]

def page_by_cursor(after, size=3):          # "older than the last one I saw"
    start = feed.index(after) + 1 if after else 0
    return feed[start:start + size]

p1 = page_by_offset(0)
print("offset page 1:", p1)
feed[:0] = ["post12", "post11"]             # 2 new posts while Priya scrolls
print("offset page 2:", page_by_offset(3))

feed = [f"post{i}" for i in range(10, 0, -1)]
c1 = page_by_cursor(None)
print("cursor page 1:", c1)
feed[:0] = ["post12", "post11"]
print("cursor page 2:", page_by_cursor(c1[-1]))

Output:

offset page 1: ['post10', 'post9', 'post8']
offset page 2: ['post9', 'post8', 'post7']
cursor page 1: ['post10', 'post9', 'post8']
cursor page 2: ['post7', 'post6', 'post5']
  • Offset pagination says "give me items 3 to 5". While Priya read page 1, two new posts arrived at the top and pushed everything down by two, so page 2 shows post9 and post8 again.
  • Cursor pagination says "give me items older than the last one I saw" (post8). New posts at the top don't matter: page 2 is post7, post6, post5. No repeats, no gaps.
  • A cursor is a bookmark that points at an item, not a position. In a real feed it's the time or id of the last post, sent back by the app with each request.

For a ranked feed, the server ranks once when you open the app, keeps that ordered list for your session, and the cursor points into it; new posts show up the next time you pull to refresh (the mini project does exactly this).

Photos: object storage + CDN

The feed never stores photos. Each post's image is uploaded once to object storage (a service that stores files by name, like Amazon S3; episode 21) and served through a CDN (content delivery network: servers near users that keep copies of popular files; episode 21). The feed holds only post ids; the app fetches the post details and then the image URLs from the CDN.

The full design

 post service ──▶ message queue ──▶ fan-out workers ──▶ feed cache (post ids per user)
      │                                                       │
 object storage + CDN (photos)              feed service: ready feed + stars → rank → page by cursor
  1. Post: the post service saves the post (and the photo to object storage), then drops a "new post" event on a message queue (episode 16).
  2. Fan-out workers take events from the queue and push the post id into each follower's feed in the feed cache, capped at a few hundred ids. Stars (say, more than a million followers) are skipped.
  3. Open: the feed service reads the user's ready feed, pulls the posts of the stars they follow, merges, ranks, and returns the first page with a cursor.
  4. Inactive users (no open in a month) are skipped by the workers; their feed is built by pulling when they come back.

A worker is a background program that takes jobs from a queue and does them, so the user who posted doesn't wait for 10,000 feed writes.

The whole chain

  1. 500 queries per open → fan-out on write.
  2. A star: 100 million writes → pull stars at read time, merge.
  3. Newest first buries Neha → rank by a score.
  4. Offset pagination repeats posts → cursor.
  5. Photos → object storage + CDN.
  6. Queue + workers + feed cache + feed service.

Common mistakes (and the real outputs)

Mistake What you see Why Fix
Query every followed account on open queries: 500 feed built in 1.1 s All the work at read time, every open Pre-build feeds: fan-out on write
Fan out every post, even a star's star posts -> 100,000,000 writes, last fan waits 100 s One post becomes 100 million writes Pull stars at read time and merge
Newest first only sister_neha last Recency is the only signal Rank with closeness, interest and recency
feed[offset:offset + size] offset page 2: ['post9', 'post8', 'post7'] New posts shift every position Cursor: posts older than the last one seen
Photos stored in the feed Huge feed rows, slow opens Images are megabytes; ids are bytes Object storage + CDN; feed holds ids
Fan-out inside the post request Posting takes seconds for big accounts The poster waits for every follower write Queue + background workers

Mini project: a feed service in about 90 lines

Fan-out on write, stars pulled at read time, ranking, and cursor pagination over a per-session ranked snapshot. To keep it small, a "star" here is anyone with more than 3 followers.

# Mini project: a feed service with fan-out on write, stars pulled at
# read time, ranking, and cursor pagination.
import heapq
from collections import defaultdict

STAR = 3                                  # tiny for the demo: >3 followers
FEED_CAP = 500

class FeedService:
    def __init__(self):
        self.followers = defaultdict(set)     # author -> followers
        self.following = defaultdict(set)     # user -> authors
        self.posts = {}                       # post id -> post
        self.by_author = defaultdict(list)    # author -> [(time, id)]
        self.feeds = defaultdict(list)        # user -> [(time, id)] ready
        self.closeness = defaultdict(float)   # (user, author) -> 0..1
        self.snapshot = {}                    # user -> ranked list
        self.writes = 0

    def follow(self, user, author, closeness=0.2):
        self.followers[author].add(user)
        self.following[user].add(author)
        self.closeness[(user, author)] = closeness

    def is_star(self, author):
        return len(self.followers[author]) > STAR

    def post(self, author, pid, t, text, likes=0.0):
        self.posts[pid] = {"id": pid, "author": author, "t": t,
                           "text": text, "likes": likes}
        self.by_author[author].insert(0, (t, pid))
        if self.is_star(author):
            return                        # stars: no fan-out
        for f in self.followers[author]:  # fan-out on write
            self.feeds[f].insert(0, (t, pid))
            del self.feeds[f][FEED_CAP:]
            self.writes += 1

    def candidates(self, user):
        stars = [a for a in self.following[user] if self.is_star(a)]
        lists = [self.feeds[user]] + [self.by_author[s] for s in stars]
        return [pid for _, pid in heapq.merge(*lists, reverse=True)]

    def score(self, user, pid, now):
        p = self.posts[pid]
        close = self.closeness[(user, p["author"])]
        return (close * 2 + p["likes"] * 10) / (now - p["t"] + 2) ** 1.5

    def page(self, user, now, cursor=None, size=3):
        if cursor is None:                # a fresh open: rank once and
            self.snapshot[user] = sorted(  # keep this order while scrolling
                self.candidates(user),
                key=lambda pid: self.score(user, pid, now), reverse=True)
        ranked = self.snapshot[user]
        start = ranked.index(cursor) + 1 if cursor else 0
        items = ranked[start:start + size]
        return items, (items[-1] if items else None)

fs = FeedService()
for fan in ["priya", "rahul", "neha", "amit", "kavya"]:
    fs.follow(fan, "star_kabir")          # 5 followers: a "star"
fs.follow("priya", "sister_neha", 1.0)
fs.follow("priya", "chai_point", 0.3)
fs.follow("priya", "meme_page", 0.1)

fs.post("meme_page", "m1", 9, "funny cat", likes=0.02)
fs.post("sister_neha", "n1", 3, "Got the job!", likes=0.3)
fs.post("chai_point", "c1", 8, "masala chai offer", likes=0.1)
fs.post("star_kabir", "k1", 7, "new film trailer", likes=0.2)
fs.post("meme_page", "m2", 10, "another meme", likes=0.01)

print("fan-out writes:", fs.writes, "(the star's post: 0)")
items, cur = fs.page("priya", now=10)
print("page 1:", [fs.posts[i]["text"] for i in items])
fs.post("meme_page", "m3", 10, "breaking meme", likes=0.01)   # arrives
items, cur = fs.page("priya", now=10, cursor=cur)
print("page 2:", [fs.posts[i]["text"] for i in items])
items, cur = fs.page("priya", now=11)                       # pull to refresh
print("refresh:", [fs.posts[i]["text"] for i in items])

Output:

fan-out writes: 4 (the star's post: 0)
page 1: ['new film trailer', 'masala chai offer', 'Got the job!']
page 2: ['another meme', 'funny cat']
refresh: ['new film trailer', 'Got the job!', 'masala chai offer']
  • The star's trailer cost 0 fan-out writes but still shows up first: it was pulled at read time.
  • Page 2 continues from the cursor in the same ranked order, even though a new meme arrived while Priya scrolled; it appears only after a refresh, and only if it ranks high enough (here it doesn't make the top 3).

Try extending it: skip fan-out for users who haven't opened the app recently, and build their feed by pulling when they return.

Practice

  1. Homework: in the PULL demo, follow 2,000 accounts. How long does one open take now?
  2. Homework: in the RANK demo, make old posts sink faster. Where does Neha land?
  3. Homework: in the PAGE demo, add 5 new posts while Priya reads page 1. What does offset page 2 show?
  4. Homework: where should Stories (posts that disappear after 24 hours) go in this design?
  5. Priya follows 3 stars and 497 normal accounts. How many lists does one open read?
  6. Why do fan-out workers sit behind a queue instead of running inside the "post" request?
  7. Why cap each ready feed at a few hundred ids?
  8. A user with 900,000 followers posts. With a star limit of 1 million, what happens? What if they gain 200,000 followers tomorrow?
Solutions and hints

1. Change range(500) to range(2000) (and 500 in the last line to 2000). Output:

queries: 2000  feed built in 4.2 s
10 million opens a minute: 333,333,333 queries/s

Four times the follows, about four times the time. Fan-out on read grows with how many accounts you follow, on every single open.

2. Change ** 1.5 to ** 3. Output:

newest first: ['news_channel', 'random_meme_page', 'college_friend', 'chai_point', 'sister_neha']
ranked:       ['college_friend', 'news_channel', 'random_meme_page', 'chai_point', 'sister_neha']

Neha drops to last again: recency now dominates and her six-hour-old post can't win. Tuning this balance is exactly what ranking teams do (with data, not guesses).

3. Replace the first feed[:0] = ["post12", "post11"] with five new posts. Output starts:

offset page 1: ['post10', 'post9', 'post8']
offset page 2: ['post12', 'post11', 'post10']

Page 2 now shows two new posts above where Priya was, and post10 again. The cursor version still gives post7, post6, post5.

4. Stories are short-lived and viewed in a tray at the top, so they use the same idea at smaller scale: a per-user list of story ids from followees (pushed for normal accounts, pulled for stars), with a 24-hour TTL so they expire on their own; the media sits in object storage behind the CDN.

5. Four: her own ready feed plus the three stars' lists.

6. A big account may need thousands of feed writes; the poster shouldn't wait for them, and if a write fails it should be retried. A queue lets the post return immediately and lets many workers share the job (episode 16).

7. Nobody scrolls through 10,000 posts, and memory per user stays small and predictable. Older posts can be fetched by pulling if someone really scrolls that far.

8. Below the limit, it's fanned out: 900,000 writes. Tomorrow they become a star: new posts are no longer pushed and are pulled instead. Posts already pushed stay in feeds; the merge still works because both paths carry the post time.

Quick quiz

Cricket star: 80 million followers
Posts: 3 photos a day
Should we push each post into every fan's feed?
  • A. Yes, fan-out on write
  • B. No: store once, fans pull it on open
  • C. Email every fan
  • D. Only push to people online
Answer

B. Pushing would be 80 million writes per photo. Store the post once in the star's list and merge it into each fan's feed when they open the app: the hybrid feed.

Interview questions: "Design the Instagram (or Twitter) feed"

1. Clarify requirements. Home feed from followees; how fresh? Ranked or chronological? Media? Scale: daily users, follows per user, max followers? Then state: open in under ~200 ms, new posts visible within seconds, infinite scroll without repeats, reads >> writes, some accounts with tens of millions of followers.

2. Estimate. "10 million feed opens a minute. Fan-out on read at 500 follows = 83 million queries a second. Fan-out on write = 1 read per open, but a 100-million-follower post = 100 million writes."

3. Data. Posts table (id, author, time, caption, media URL); follow graph (follower → followee, both directions); feed cache (user → list of post ids, capped); media in object storage behind a CDN.

4. Feed generation. Hybrid: fan-out on write via queue + workers for normal accounts; stars pulled and merged at read time; skip inactive users.

5. Ranking. Candidates from the hybrid feed → features (closeness, engagement, recency, media type) → a learned model scores each → sort. Keep the ranked list per session.

6. Pagination. Cursor-based, never offset.

7. Scale and failure. Shard the feed cache by user id; replicate the follow graph; if ranking is slow or down, fall back to chronological (graceful degradation).

Common follow-ups:

  • Push vs pull: when would you choose each? (Push when followers are few and reads dominate; pull for accounts with huge follower counts or for inactive users.)
  • How do you choose the star threshold? (By measured cost: where one post's fan-out time and writes exceed what you accept; it can be dynamic.)
  • What happens when someone unfollows? (Stop future fan-out; filter their posts out at read time; optionally clean the cached feed in the background.)
  • How do you handle deleted posts? (Feeds hold ids; the feed service drops ids whose post is gone when it loads details.)
  • How do you add ads or suggested posts? (Mix them into the candidate list before ranking, with their own rules.)

Glossary

Term Meaning
Feed A list of posts chosen for one user to scroll, e.g. the Instagram home feed
Post One shared item: photo/video, caption, author, time
Follow / follower / followee Asking to see someone's posts / someone who follows you / someone you follow
Query One question sent to a database
Reads vs writes Reading data (opening a feed) vs changing it (posting); feeds have far more reads
Fan-out One thing spreading out to many places
Fan-out on read (pull) Build the feed when the user opens it, by querying every followee
Fan-out on write (push) When someone posts, add it to every follower's ready-made feed
Ready-made (precomputed) feed A stored list of post ids per user, kept up to date by fan-out on write
Feed cache The fast store (e.g. Redis) holding every user's ready-made feed
Celebrity (hot-key) problem A few accounts with huge follower counts make fan-out on write explode
Hybrid feed Push for normal accounts, pull for stars, merged at read time
defaultdict A Python dict that creates a default value (e.g. an empty list) for new keys
deque(maxlen=...) A Python list that drops its oldest items beyond a maximum length
heapq.merge Merges already-sorted lists into one sorted list
Ranking Ordering items by a score of how much the user will care
Candidate A post considered for the feed before ranking
Signal / feature One input to the score, e.g. closeness, likes per view, age
Pagination Loading a long list one page at a time
Offset pagination "Items N to N+size": breaks when new items arrive
Cursor pagination "Items after this bookmark": stable while new items arrive
Cursor A bookmark pointing at the last item seen
Object storage A service storing files by name, e.g. Amazon S3 (episode 21)
CDN Servers near users that keep copies of popular files (episode 21)
Message queue A list of jobs one service drops in and others take out (episode 16)
Worker A background program that takes jobs from a queue and does them
Graceful degradation Keep the core working with a simpler result when a part fails

Summary

  • Feeds are read far more than written, so the read path must be cheap.
  • Fan-out on read took 500 queries and about a second per open (83 million queries a second at scale).
  • Fan-out on write pre-builds a feed per user: one read per open.
  • A star with 100 million followers turns one post into 100 million writes; the hybrid feed pushes normal accounts and pulls stars at read time, merged by time.
  • Newest first buries what matters (Instagram, 2016: people missed 70 percent of their feeds); rank candidates with a score of interest, closeness and recency.
  • Offset pagination repeats posts when new ones arrive; cursor pagination doesn't.
  • Photos live in object storage behind a CDN; feeds hold only ids.
  • The design: post service → queue → fan-out workers → feed cache → feed service (merge, rank, page).

Next episode: an IPL final, with tens of millions of people watching the same ball live. How do you stream that without falling over? Design Hotstar live streaming.

Sources: Instagram, "See Posts You Care About First in Your Feed" (about.instagram.com, 15 March 2016): "people miss on average 70 percent of their feeds"; "The order of photos and videos in your feed will be based on the likelihood you'll be interested in the content", with relationship and timeliness as further signals. All sizes in the demos are our own stated assumptions.