SyntaxPilot
Courses/Low-Level Design/Lesson 69
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Low-Level Design · Lesson 69 of 90

Design a pub-sub messaging system

publish-subscribeTopicPublisherSubscriberMessageBrokerpartitionsoffsetsconsumer groupspush vs pull deliveryat-most-onceat-least-onceexactly-oncemessage retention

These are the notes for Low-Level Design (LLD) episode 69: designing a publish-subscribe (pub-sub) messaging system, the idea behind message brokers that large applications use so their parts can talk without knowing each other. We start with one service calling three others directly, watch it crash, and fix one problem at a time: a Broker with Topics, push vs pull delivery, a topic stored as a log with offsets and message retention, partitions, consumer groups and concurrent consumers, and the three delivery guarantees: at-most-once, at-least-once and exactly-once.

How the code was run: every program was saved as Main.java in a fresh folder and run with javac Main.java && java -cp . Main on OpenJDK 21. Every output below is the real output, pasted exactly (millisecond timings change from run to run). Orders and services are made up for teaching.

1. The problem: the newspaper vendor

Newspapers Pub-sub
The newspaper office prints the paper; it doesn't know every reader A publisher sends messages without knowing who reads them
The vendor picks up every paper and delivers to each subscriber The broker: the middleman that stores and delivers
Each home chooses its papers: an English daily, a sports weekly Subscribers subscribe to topics
On holiday? The vendor keeps a week of papers, not a year Message retention

2. The words

  • Message: one piece of news, e.g. order 742 paid.
  • Publisher (producer): sends messages to a topic; doesn't know who reads them.
  • Topic: a named channel of messages, e.g. orders, payments.
  • Subscriber (consumer): receives a topic's messages.
  • Broker: the middleman that stores messages and delivers them.
  • Publish-subscribe: the whole arrangement: publishers and subscribers only know topics, never each other.

3. Attempt 1: call everyone directly

class EmailService     { void send(int order)  { System.out.println("  email: order " + order + " confirmed"); } }
class SmsService       { void send(int order)  { System.out.println("  sms: order " + order + " confirmed"); } }
class AnalyticsService {
    boolean down = false;
    void track(int order) { if (down) throw new IllegalStateException("analytics is down");
                            System.out.println("  analytics: order " + order + " counted"); } }
class OrderService {                                  // attempt: the order service calls everyone itself
    EmailService email = new EmailService();  SmsService sms = new SmsService();
    AnalyticsService analytics = new AnalyticsService();
    void place(int order) {
        System.out.println("order " + order + " paid");
        email.send(order);                            // a new listener? edit THIS method again
        sms.send(order);
        analytics.track(order);                       // and wait for every one of them
        System.out.println("order " + order + " DONE");
    }
}
public class Main {
    public static void main(String[] args) {
        OrderService orders = new OrderService();
        orders.place(741);
        orders.analytics.down = true;                 // a reporting service has a bad minute
        orders.place(742);
    }
}

Real output:

order 741 paid
  email: order 741 confirmed
  sms: order 741 confirmed
  analytics: order 741 counted
order 741 DONE
order 742 paid
  email: order 742 confirmed
  sms: order 742 confirmed
Exception in thread "main" java.lang.IllegalStateException: analytics is down
    at AnalyticsService.track(Main.java:5)
    at OrderService.place(Main.java:14)
    at Main.main(Main.java:23)

Order 742 was paid, the email and SMS went out, then analytics threw IllegalStateException: analytics is down and the program died: order 742 never reached DONE. A reporting service broke checkout.

Why it failed

  • Tight coupling (episode 10): OrderService knows every listener by name.
  • A new listener means editing OrderService again (breaks Open-Closed).
  • It waits for each one, in turn, and one failure stops all.

4. A broker with push delivery

The publisher only says publish("orders", msg) and carries on. The broker keeps a list of subscribers per topic. Push delivery: the broker calls each subscriber right away. Each call has its own try/catch, so one broken subscriber can't stop the others.

  • Consumer<Message>: a Java functional interface for "a function that takes a Message and returns nothing", so a subscriber can be a lambda: m -> System.out.println(m.body()).
import java.util.*;  import java.util.function.Consumer;
record Message(String topic, String body) {}
class Broker {                                            // the middleman: publishers never see subscribers
    private final Map<String, List<Consumer<Message>>> subscribers = new HashMap<>();
    void subscribe(String topic, Consumer<Message> s) {
        subscribers.computeIfAbsent(topic, t -> new ArrayList<>()).add(s); }
    void publish(String topic, String body) {             // PUSH: the broker calls each subscriber now
        Message m = new Message(topic, body);
        for (Consumer<Message> s : subscribers.getOrDefault(topic, List.of())) {
            try { s.accept(m); }
            catch (Exception e) { System.out.println("  (lost for one subscriber: " + e.getMessage() + ")"); } } }
}
public class Main {
    static boolean analyticsDown = false;  static List<String> seen = new ArrayList<>();
    public static void main(String[] args) {
        Broker broker = new Broker();
        broker.subscribe("orders", m -> System.out.println("  email: " + m.body()));
        broker.subscribe("orders", m -> System.out.println("  sms: " + m.body()));
        broker.subscribe("orders", m -> {
            if (analyticsDown) throw new IllegalStateException("analytics is down");
            seen.add(m.body());  System.out.println("  analytics: " + m.body()); });
        broker.publish("orders", "order 741 paid");
        analyticsDown = true;   broker.publish("orders", "order 742 paid");
        System.out.println("order 742 DONE");                // the order service is safe now
        analyticsDown = false;  broker.publish("orders", "order 743 paid");
        System.out.println("analytics saw " + seen);
    }
}

Real output:

  email: order 741 paid
  sms: order 741 paid
  analytics: order 741 paid
  email: order 742 paid
  sms: order 742 paid
  (lost for one subscriber: analytics is down)
order 742 DONE
  email: order 743 paid
  sms: order 743 paid
  analytics: order 743 paid
analytics saw [order 741 paid, order 743 paid]

The failure stayed inside the broker (order 742 DONE), but analytics never saw order 742: with push, if you're not there at that moment, the message is gone.

5. Keep the messages: a log, offsets, pull, retention

  • Store each topic as an append-only log: a list you only add to the end of. Messages are kept after delivery.
  • Offset: a message's position in the log: 0, 1, 2, ...
  • Pull delivery: the subscriber asks the broker "give me everything from offset 5". The subscriber decides when, and how many.
  • Each subscriber keeps its own offset, like a bookmark. After being down, it pulls from its bookmark and catches up.
  • Message retention: the broker deletes old messages after some time or beyond some count; a subscriber that's down longer than that misses them.
Push Pull
Who starts delivery Broker calls the subscriber Subscriber asks the broker
Speed Immediate When the subscriber polls
A slow or offline subscriber Misses messages, or the broker must buffer Reads later from its offset
Flow control Broker can overwhelm the subscriber Subscriber takes what it can handle
import java.util.*;
record Message(long offset, String body) {}
class Topic {                                             // an append-only log: messages are KEPT
    private final List<Message> log = new ArrayList<>();  private long next = 0;  private final int retention;
    Topic(int retention) { this.retention = retention; }
    synchronized void append(String body) {
        log.add(new Message(next++, body));               // offset = the message's position: 0, 1, 2, ...
        if (log.size() > retention) log.remove(0); }      // retention: keep only the newest N
    synchronized List<Message> read(long from) {          // PULL: the subscriber asks, from ITS offset
        return log.stream().filter(m -> m.offset() >= from).toList(); }
    synchronized long oldest() { return log.isEmpty() ? next : log.get(0).offset(); }
}
class Subscriber {  final String name;  long offset = 0;        // where THIS subscriber is up to
    Subscriber(String name) { this.name = name; }
    void poll(Topic t) {
        if (offset < t.oldest()) { System.out.println("  " + name + ": #" + offset + " to #" + (t.oldest() - 1)
                                       + " were deleted before I read them!");  offset = t.oldest(); }
        for (Message m : t.read(offset)) {
            System.out.println("  " + name + " got #" + m.offset() + " " + m.body());  offset = m.offset() + 1; } }
}
public class Main { public static void main(String[] args) {
    Topic orders = new Topic(5);                                     // retention: the newest 5
    Subscriber email = new Subscriber("email"), analytics = new Subscriber("analytics");
    orders.append("order 741 paid");  email.poll(orders);  analytics.poll(orders);
    orders.append("order 742 paid");  email.poll(orders);            // analytics is down: no poll
    orders.append("order 743 paid");  email.poll(orders);
    System.out.println("analytics is back:");  analytics.poll(orders);      // catches up from offset 1
    for (int o = 744; o <= 750; o++) orders.append("order " + o + " paid");   // analytics down a long time
    System.out.println("much later:");  analytics.poll(orders); } }

Real output:

  email got #0 order 741 paid
  analytics got #0 order 741 paid
  email got #1 order 742 paid
  email got #2 order 743 paid
analytics is back:
  analytics got #1 order 742 paid
  analytics got #2 order 743 paid
much later:
  analytics: #3 to #4 were deleted before I read them!
  analytics got #5 order 746 paid
  analytics got #6 order 747 paid
  analytics got #7 order 748 paid
  analytics got #8 order 749 paid
  analytics got #9 order 750 paid

Back after a short outage, analytics read offsets 1 and 2: nothing lost. After a long outage, #3 to #4 were deleted before I read them!: retention is a promise with a time limit.

6. Partitions and consumer groups

One consumer doing 300 messages × 2 ms each is slow. Two ideas fix it:

  • Partition: a topic split into several logs. A message goes to partition = hash(key) % n. The same key always lands in the same partition, so messages for one order (or one customer) stay in order. There is no order between partitions.
  • Consumer group: a set of consumers that share one topic's work. Each partition is given to exactly one consumer in the group. Different groups are independent: every group gets every message.
  • Concurrent consumers: the consumers of a group run at the same time, on different threads.
  • More consumers than partitions → the extra consumers are idle: partitions limit the parallelism.
  • Math.floorMod(a, n): a remainder that is never negative (a hash code can be negative; % would then give a negative partition number).
Post office Pub-sub
Letters sorted by PIN code into 3 bags; same PIN, same bag Partitions by key
A team of 3 postmen, one bag each A consumer group
The census office gets copies of everything with its own team Another consumer group
A 4th postman with no bag waits An idle consumer
import java.util.*;  import java.util.concurrent.*;
record Message(int key, String body) {}
class Topic {                                          // one topic, split into partitions
    final List<List<Message>> partitions = new ArrayList<>();
    Topic(int n) { for (int i = 0; i < n; i++) partitions.add(new ArrayList<>()); }
    void publish(int key, String body) {               // same key -> same partition -> kept in order
        partitions.get(Math.floorMod(Integer.hashCode(key), partitions.size())).add(new Message(key, body)); }
}
class ConsumerGroup {                                  // shares the work: each partition has ONE consumer
    final String name;  final int size;
    ConsumerGroup(String name, int size) { this.name = name;  this.size = size; }
    void consume(Topic t) throws Exception {
        Map<String, Integer> done = new ConcurrentSkipListMap<>();
        for (int c = 0; c < size; c++) done.put(name + "-" + c, 0);
        ExecutorService threads = Executors.newFixedThreadPool(size);   // concurrent consumers
        long start = System.nanoTime();
        for (int p = 0; p < t.partitions.size(); p++) {
            String consumer = name + "-" + (p % size);  List<Message> part = t.partitions.get(p);
            threads.submit(() -> { for (Message m : part) {           // pretend: 2 ms of work each
                Thread.sleep(2);  done.merge(consumer, 1, Integer::sum); }  return null; }); }
        threads.shutdown();  threads.awaitTermination(1, TimeUnit.MINUTES);
        System.out.printf("%-9s %s in %d ms%n", name, done, (System.nanoTime() - start) / 1_000_000); }
}
public class Main { public static void main(String[] args) throws Exception {
    Topic orders = new Topic(3);                       // 3 partitions
    for (int o = 1; o <= 300; o++) orders.publish(o, "order " + o + " paid");
    new ConsumerGroup("analytics", 1).consume(orders);      // each GROUP gets every message
    new ConsumerGroup("email", 3).consume(orders);
    new ConsumerGroup("sms", 4).consume(orders); } }

Real output:

analytics {analytics-0=300} in 654 ms
email     {email-0=100, email-1=100, email-2=100} in 214 ms
sms       {sms-0=100, sms-1=100, sms-2=100, sms-3=0} in 212 ms

One consumer: ~700 ms. Three consumers: 100 each in roughly a third of the time. Four consumers on three partitions: sms-3=0.

7. Delivery guarantees

A consumer commits its offset: it saves in the broker "done up to here". After a crash, it restarts from its committed offset. The order of "do the work" and "commit" decides the guarantee:

Guarantee Order A crash in between Good for
At-most-once Commit, then work The message is lost Metrics where losing one is fine
At-least-once Work, then commit The message is done twice Most systems, with idempotent consumers
Exactly-once (in effect) At-least-once + an idempotent consumer The repeat is detected and skipped Payments, emails, anything that must not repeat

Idempotent: doing it twice has the same effect as doing it once. Example: remember the ids of handled messages and skip a repeat. In a real system that set of ids must be saved together with the work (for example in the same database transaction), or a crash could lose it.

import java.util.*;
enum Mode { AT_MOST_ONCE, AT_LEAST_ONCE, EXACTLY_ONCE }
class EmailConsumer {
    static final List<String> TOPIC = List.of("m0", "m1", "m2", "m3");    // message ids
    int committed = 0;                                // the offset saved in the broker: "I'm done up to here"
    final List<String> sent = new ArrayList<>();  final Set<String> seenIds = new HashSet<>();
    final Mode mode;  EmailConsumer(Mode mode) { this.mode = mode; }
    void send(String id) {
        if (mode == Mode.EXACTLY_ONCE && !seenIds.add(id)) return;     // idempotent: already done, skip
        sent.add(id); }
    void run(int crashAt) {                           // crashAt = -1 means: no crash this time
        for (int off = committed; off < TOPIC.size(); off++) {
            String id = TOPIC.get(off);
            if (mode == Mode.AT_MOST_ONCE) {          // commit FIRST, then work
                committed = off + 1;  if (off == crashAt) return;  send(id); }
            else {                                    // work FIRST, then commit
                send(id);  if (off == crashAt) return;  committed = off + 1; } } }
}
public class Main { public static void main(String[] args) {
    for (Mode mode : Mode.values()) {
        EmailConsumer c = new EmailConsumer(mode);
        c.run(2);                                      // the consumer crashes in the middle of m2
        c.run(-1);                                     // it restarts from its committed offset
        long lost = EmailConsumer.TOPIC.stream().filter(id -> !c.sent.contains(id)).count();
        long twice = c.sent.size() - new HashSet<>(c.sent).size();
        System.out.printf("%-13s emails %s  lost %d, duplicates %d%n", mode, c.sent, lost, twice); } } }

Real output:

AT_MOST_ONCE  emails [m0, m1, m3]  lost 1, duplicates 0
AT_LEAST_ONCE emails [m0, m1, m2, m2, m3]  lost 0, duplicates 1
EXACTLY_ONCE  emails [m0, m1, m2, m3]  lost 0, duplicates 0

8. The class diagram

Publisher ──publish(topic, key, body)──▶ Broker
Broker: topics by name ─▶ Topic ─▶ partitions[ ]   partition = hash(key) % n
          each partition: an append-only log, offsets 0, 1, 2, ...  (retention: newest N)
Subscriber / Consumer: PULL read(partition, from = its offset), then commit the offset
ConsumerGroup: each partition → exactly ONE consumer of the group; every group gets all
Delivery: commit then work = at-most-once · work then commit = at-least-once
          at-least-once + idempotent consumer (skip seen ids) = effectively exactly-once

Patterns used: Observer (episode 35) is the in-process ancestor of pub-sub; here a broker (a Mediator, episode 41) sits between publishers and subscribers, and the subscriber is a Strategy (a Consumer<Message> function).

9. Real systems: Apache Kafka's words

Apache Kafka uses exactly these ideas. From its introduction:

  • "A topic is a category or feed name to which records are published."
  • "Each partition is an ordered, immutable sequence of records that is continually appended to–a structured commit log." Records are "assigned a sequential id number called the offset that uniquely identifies each record within the partition."
  • The cluster keeps published records for a "configurable retention period".
  • "each record published to a topic is delivered to one consumer instance within each subscribing consumer group."
  • "Kafka only provides a total order over records within a partition, not between different partitions in a topic."

10. Interview follow-ups

Follow-up Answer
Survive a broker crash Write each partition's log to disk; keep copies (replicas) on other machines
Consumers falling behind Measure lag = newest offset − committed offset, per partition; add consumers (up to the partition count)
A message that always fails After N tries, move it to a dead-letter topic so it doesn't block the partition (mini project)
Keep order for one customer Use the customer id as the key: same key → same partition → in order
A consumer joins or leaves a group Rebalance: hand its partitions to the remaining consumers; they continue from the committed offsets
Push or pull? Pull lets slow consumers go at their own pace and catch up; push is lower latency
Message too big Store the payload elsewhere and publish a reference

11. Common mistakes

Mistake What you see (real) Why Fix
Calling every listener directly Exception in thread "main" java.lang.IllegalStateException: analytics is down Tight coupling Publish to a topic
Push only, nothing stored analytics saw [order 741 paid, order 743 paid] Missed while down A log + offsets, pull
Retention shorter than downtime analytics: #3 to #4 were deleted before I read them! Old messages expire Longer retention; watch lag
More consumers than partitions sms-3=0 1 partition → 1 consumer per group More partitions
Commit before the work lost 1 At-most-once Work, then commit; idempotent consumer
Publishing to a topic that doesn't exist java.lang.NullPointerException: Cannot invoke "java.util.List.add(Object)" because the return value of "java.util.Map.get(Object)" is null "order" vs "orders" Check the topic and throw a clear error
Counting inside a subscriber lambda with a local int error: local variables referenced from a lambda expression must be final or effectively final Lambdas can't change local variables A field, an AtomicInteger, or a list
Forgetting the import error: cannot find symbol ... symbol: class Consumer It lives in java.util.function import java.util.function.Consumer;

Mini project: an in-memory broker

Partitions by key, retention, committed offsets per group and partition, lag, at-least-once delivery with an idempotent consumer, retries and a dead-letter topic after 3 failed tries.

import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Consumer;

// Mini project: an in-memory broker with partitions, retention, per-group committed offsets, lag,
// at-least-once delivery with an idempotent consumer, and a dead-letter topic after 3 failed tries.
record Record(String topic, int partition, long offset, String key, String value) {}

class Partition {
    private final List<Record> log = new ArrayList<>();
    private long next = 0;
    private final int retention;
    Partition(int retention) { this.retention = retention; }
    synchronized long append(String topic, int p, String key, String value) {
        log.add(new Record(topic, p, next, key, value));
        if (log.size() > retention) log.remove(0);
        return next++;
    }
    synchronized List<Record> read(long from, int max) {
        List<Record> out = new ArrayList<>();
        for (Record r : log) if (r.offset() >= from && out.size() < max) out.add(r);
        return out;
    }
    synchronized long oldest() { return log.isEmpty() ? next : log.get(0).offset(); }
    synchronized long end() { return next; }
}

class Broker {
    private final Map<String, List<Partition>> topics = new ConcurrentHashMap<>();
    private final Map<String, Long> committed = new ConcurrentHashMap<>();   // "group|topic|p" -> next offset to read
    void createTopic(String name, int partitions, int retention) {
        List<Partition> ps = new ArrayList<>();
        for (int i = 0; i < partitions; i++) ps.add(new Partition(retention));
        topics.put(name, ps);
    }
    int partitions(String topic) { return topics.get(topic).size(); }
    Record publish(String topic, String key, String value) {
        List<Partition> ps = topics.get(topic);
        int p = Math.floorMod(key.hashCode(), ps.size());               // same key -> same partition
        long off = ps.get(p).append(topic, p, key, value);
        return new Record(topic, p, off, key, value);
    }
    List<Record> poll(String group, String topic, int p, int max) {
        Partition part = topics.get(topic).get(p);
        long from = committed.getOrDefault(group + "|" + topic + "|" + p, 0L);
        if (from < part.oldest()) {
            System.out.println("  [" + group + "] p" + p + ": offsets " + from + ".." + (part.oldest() - 1) + " expired");
            from = part.oldest();
        }
        return part.read(from, max);
    }
    void commit(String group, Record r) { committed.put(group + "|" + r.topic() + "|" + r.partition(), r.offset() + 1); }
    long lag(String group, String topic) {
        long lag = 0;
        for (int p = 0; p < partitions(topic); p++)
            lag += topics.get(topic).get(p).end() - committed.getOrDefault(group + "|" + topic + "|" + p, 0L);
        return lag;
    }
}

class GroupConsumer {                       // at-least-once: work first, then commit
    private final Broker broker;  private final String group, topic;
    private final Consumer<Record> handler;
    private final Set<String> done = new HashSet<>();               // idempotency: ids already handled
    private final Map<String, Integer> failures = new HashMap<>();
    GroupConsumer(Broker broker, String group, String topic, Consumer<Record> handler) {
        this.broker = broker;  this.group = group;  this.topic = topic;  this.handler = handler; }
    void pollOnce(int p) {
        for (Record r : broker.poll(group, topic, p, 10)) {
            String id = r.topic() + "-" + r.partition() + "-" + r.offset();
            if (done.contains(id)) { broker.commit(group, r);  continue; }   // a repeat: skip the work
            try {
                handler.accept(r);
                done.add(id);
                broker.commit(group, r);
            } catch (RuntimeException e) {
                int n = failures.merge(id, 1, Integer::sum);
                if (n < 3) { System.out.println("  [" + group + "] " + r.value() + " failed (try " + n + "), will retry");  return; }
                broker.publish(topic + ".dead", r.key(), r.value());        // give up: park it
                System.out.println("  [" + group + "] " + r.value() + " -> dead-letter topic after 3 tries");
                broker.commit(group, r);
            }
        }
    }
}

public class Main {
    public static void main(String[] args) {
        Broker broker = new Broker();
        broker.createTopic("orders", 2, 4);
        broker.createTopic("orders.dead", 1, 100);
        for (int o = 741; o <= 746; o++) {
            Record r = broker.publish("orders", "customer" + (o % 3), "order " + o + " paid");
            System.out.println("published " + r.value() + " -> p" + r.partition() + " offset " + r.offset());
        }
        List<String> emails = new ArrayList<>();
        GroupConsumer email = new GroupConsumer(broker, "email", "orders", r -> {
            if (r.value().equals("order 744 paid")) throw new IllegalStateException("bad address");
            emails.add(r.value()); });
        System.out.println("email lag before: " + broker.lag("email", "orders"));
        for (int round = 0; round < 3; round++)
            for (int p = 0; p < broker.partitions("orders"); p++) email.pollOnce(p);
        System.out.println("emails sent: " + emails);
        System.out.println("email lag after: " + broker.lag("email", "orders"));
        System.out.println("dead letters: " + broker.poll("ops", "orders.dead", 0, 10).stream().map(Record::value).toList());
        for (int o = 747; o <= 756; o++) broker.publish("orders", "customer" + (o % 3), "order " + o + " paid");
        System.out.println("analytics joins late, retention 4 per partition:");
        int seen = 0;
        for (int p = 0; p < broker.partitions("orders"); p++) seen += broker.poll("analytics", "orders", p, 100).size();
        System.out.println("analytics can still read " + seen + " of 16 messages");
    }
}

Real output:

published order 741 paid -> p0 offset 0
published order 742 paid -> p1 offset 0
published order 743 paid -> p0 offset 1
published order 744 paid -> p0 offset 2
published order 745 paid -> p1 offset 1
published order 746 paid -> p0 offset 3
email lag before: 6
  [email] order 744 paid failed (try 1), will retry
  [email] order 744 paid failed (try 2), will retry
  [email] order 744 paid -> dead-letter topic after 3 tries
emails sent: [order 741 paid, order 743 paid, order 742 paid, order 745 paid, order 746 paid]
email lag after: 0
dead letters: [order 744 paid]
analytics joins late, retention 4 per partition:
  [analytics] p0: offsets 0..6 expired
  [analytics] p1: offsets 0..0 expired
analytics can still read 8 of 16 messages

Order 744 fails 3 times and is parked in orders.dead; the rest are delivered and lag drops from 6 to 0. Analytics joins after 16 messages, but retention keeps only 4 per partition, so it can read 8.

Practice

1. Add commit(offset) per group to the Topic, so a restarted consumer continues where it left off.

Answer

Keep a map "group|topic|partition" → next offset; poll reads from it, commit(group, record) stores record.offset() + 1. See Broker.commit and Broker.poll in the mini project.

2. Print each group's lag.

Answer

For each partition: end offset − committed offset, summed. See Broker.lag: email lag before: 6, email lag after: 0.

3. Add a dead-letter topic after 3 failures.

Answer

Count failures per message id; on the 3rd, publish it to topic + ".dead" and commit past it so the partition moves on. See GroupConsumer.pollOnce: order 744 paid -> dead-letter topic after 3 tries.

4. Make Broker.publish safe for many publishers at once.

Answer

Each Partition.append is synchronized (one writer at a time per partition, so offsets are unique and in order), and the topic map is a ConcurrentHashMap. Different partitions can be written in parallel. See the mini project.

5. A consumer group has 3 consumers and the topic has 2 partitions. You need more throughput. What do you change?

Answer

Add partitions. One consumer is already idle (more consumers than partitions); only more partitions allow more parallel consumers in one group.

Quick quiz

Topic orders has 4 partitions. Group A has 2 consumers, group B has 6. How many consumers in group B get no partition? (a) 0 (b) 2 (c) 4 (d) 6

Answer

(b) 2. Each partition goes to exactly one consumer of a group: 4 partitions keep 4 of B's 6 busy. Group A's 2 consumers take 2 partitions each. Real run with 40 messages:

A         {A-0=20, A-1=20} in 51 ms
B         {B-0=10, B-1=10, B-2=10, B-3=10, B-4=0, B-5=0} in 25 ms

Interview questions

  1. What problem does pub-sub solve? Decoupling: publishers don't know subscribers; adding a subscriber doesn't change the publisher; a slow or broken subscriber doesn't block it.
  2. Push vs pull? Push: the broker calls subscribers immediately, but offline ones miss messages. Pull: subscribers read from their offset when ready, and can catch up.
  3. What is an offset? A message's position in a partition's log; each consumer group stores the next offset it will read.
  4. Why partitions? To split a topic so several consumers can work in parallel, while keeping order per key.
  5. What is a consumer group? Consumers sharing a topic's work: each partition to exactly one member; every group gets all messages.
  6. What happens with more consumers than partitions? The extra consumers are idle.
  7. At-most-once vs at-least-once? Commit before work (may lose) vs work before commit (may duplicate).
  8. How do you get exactly-once in practice? At-least-once delivery plus idempotent processing (dedupe by id, stored with the work).
  9. What is message retention, and what's the risk? Old messages are deleted after a time or size limit; a consumer down longer than that loses them.
  10. How do you handle a poison message? Retry a few times, then move it to a dead-letter topic and alert.

Glossary

Term Meaning, with an example
Publish-subscribe (pub-sub) Publishers send to topics; subscribers receive from topics; neither knows the other
Message One piece of news: order 742 paid
Publisher (producer) Sends messages: the order service
Subscriber (consumer) Receives messages: email, SMS, analytics
Topic A named channel of messages: orders
Broker The middleman that stores and delivers messages
Tight coupling One class depends directly on many others by name
Push delivery The broker calls the subscriber immediately
Pull delivery The subscriber asks the broker for messages from its offset
Consumer (Java) A function that takes a T and returns nothing: m -> print(m)
Append-only log A list you only add to the end of; old entries never change
Offset A message's position in a partition: 0, 1, 2
Commit (an offset) Save "done up to here" so a restart continues from it
Message retention How long or how many messages the broker keeps: newest 5
Partition One of several logs a topic is split into
Key The value that picks the partition: an order id, a customer id
hash(key) % n Turns a key into a partition number from 0 to n−1
Math.floorMod A remainder that is never negative
Consumer group Consumers that share a topic's work; each partition to one member
Concurrent consumers Consumers running at the same time on different threads
Idle consumer A consumer with no partition: sms-3=0
Thread pool A fixed set of threads that run submitted tasks: Executors.newFixedThreadPool(3)
At-most-once Each message is processed 0 or 1 times: commit first; may lose
At-least-once Each message is processed 1 or more times: work first; may duplicate
Exactly-once Each message has its effect exactly once: at-least-once + idempotency
Idempotent Doing it twice = doing it once: skip an id already handled
Lag How far a consumer is behind: newest offset − committed offset
Dead-letter topic Where messages that keep failing are parked: orders.dead
Rebalance Reassigning partitions when consumers join or leave a group
Replica A copy of a partition on another machine, for crashes
Apache Kafka A widely used open-source broker built on topics, partitions, offsets and consumer groups

Summary

  • Calling every listener directly coupled checkout to analytics: one failure crashed order 742.
  • A broker with topics and push delivery contained the failure, but a subscriber that was down missed order 742.
  • A topic as an append-only log with offsets and pull delivery let subscribers catch up, until retention deleted old messages.
  • Partitions (by key, ordered within a partition) and consumer groups with concurrent consumers shared the work: 100 each, about 3× faster; extra consumers sit idle.
  • Commit then work = at-most-once (lost 1); work then commit = at-least-once (1 duplicate); add an idempotent consumer for exactly-once in effect.