Concept Drills: 15 Pub-Sub Probes

Cover the answer, attempt it out loud, then compare. If your answer explains what without explaining what it costs, keep going.

1. What is the difference between a queue and a pub-sub system?

Weak: "Pub-sub sends to multiple consumers."

Strong: "In a queue, exactly one consumer processes each message; in pub-sub, every subscriber does. That single difference cascades: adding a consumer splits work in a queue and duplicates it in pub-sub, and — the part people miss — consumption can no longer drive deletion, because the system can't know when the last subscriber has read. So deletion moves to a retention policy. That's why pub-sub storage is a log with per-consumer offsets rather than a queue."

2. In what sense are producers and consumers decoupled?

Weak: "They don't talk to each other directly."

Strong: "Three specific independences. Immune to slow consumers — a subscriber taking ten seconds per message doesn't slow the publisher at all. Immune to consumer count — publishing to one costs the same as publishing to a thousand, so the publisher's work is O(1) in fan-out. Immune to consumer failure — a subscriber crashing doesn't fail the publish. And the second one is a requirement on the design, not a freebie: the queue-per-consumer design breaks it by making the publish path O(N)."

3. Why does the queue-per-consumer design fail?

Weak: "Too many queues."

Strong: "Two failures. Storage amplification: the same message is copied into every subscriber's queue — a 1 KB post to 10 million followers is 10 GB written, per post. Metadata overhead: millions of queues, each with its own state, replication, and cluster-manager entry. But the real indictment is asymptotic — publishing becomes O(N) in subscriber count, which destroys the exact property pub-sub exists to provide. It's not expensive, it's the wrong complexity class."

4. Why doesn't reference counting on a shared queue fix it?

Weak: "It's complicated to implement."

Strong: "Head-of-line blocking. A queue is served from the head. If nine of ten subscribers have consumed the head message and the tenth is slow or offline, that message can't be removed — and since there's only a head to read from, the other nine are stuck behind it. One slow subscriber freezes everyone, which makes consumers dependent on each other — arguably worse than the storage problem. The cause isn't the counter, it's the queue API: both failures come from reusing an abstraction that's wrong for this problem."

5. What three properties does the fix need?

Weak: "It needs to be more scalable."

Strong: "One copy of each message, independent reads at arbitrary positions so no consumer blocks another, and per-consumer position tracking. That specification is exactly an append-only log with offsets — and the source itself points at Linux hard links as the analogy: one copy, many independent references, deletion when nothing points at it. The crucial extension over the shared-queue design is that a hard link lets each reference access the file freely at any offset, which is precisely what a queue head cannot do."

6. Why are topics split into partitions?

Weak: "To scale horizontally."

Strong: "Specifically because disk I/O bounds a single broker. A topic on one machine is capped by that machine's disk throughput — a physical ceiling no caching removes. Partitioning is how you get more spindles onto one logical topic, and it makes partition count a capacity calculation: enough that aggregate disk throughput exceeds peak write rate, with headroom."

7. What else does partition count determine?

Weak: "How the data is spread out."

Strong: "Your maximum consumer parallelism. A partition is read as one ordered sequence, so within a consumer group one partition maps to at most one consumer — ten partitions means ten consumers, and an eleventh sits idle. It's the 'ordering domain equals parallelism domain' rule made concrete. Practical consequence: over-provision partitions, because adding them later redistributes keys and breaks any per-key ordering consumers were relying on."

8. Why segments instead of one file per partition?

Weak: "It's easier to manage."

Strong: "Deletion. Retention means removing old messages, and deleting a prefix from the middle of one enormous file is expensive and fragmenting. With a series of segment files, expiry becomes 'delete the oldest segment' — an O(1) filesystem operation with no rewriting. Segments also bound crash recovery: you validate the tail of the active segment rather than scanning terabytes, and the offset index per segment stays small. Topics are the logical unit, partitions the parallelism unit, segments the retention and recovery unit."

9. How do offsets solve head-of-line blocking?

Weak: "Each consumer knows where it is."

Strong: "Because records are immutable, a reader is just a cursor into a file, and cursors don't interfere. Consumer A sits at offset 6 and consumer B at offset 2, reading the same file simultaneously — A being slow, stopped, or offline has zero effect on B, because there's no head, no removal, and nothing to block on. Deletion happens separately and in bulk, driven by retention. Immutable log plus per-reader offset equals independent consumption — that's the whole architectural answer."

10. How do you get strict ordering, and what's the default?

Weak: "Messages are ordered within a topic."

Strong: "The default gives no ordering at all — weighted round robin puts consecutive messages from one producer on different partitions, and partitions are independent logs, so no relationship survives. Ordering is opt-in: the producer supplies a partition_ID in the write call and everything with that ID lands on one log in order. The rule is to partition by the smallest key that needs ordering — channel ID for chat, publisher ID for a feed. Kafka refines this by hashing a message key, so the client says what's related rather than where to put it."

11. Why is round robin weighted?

Weak: "To balance load."

Strong: "Because plain round robin assumes every partition can absorb the same rate, and in a real cluster that's false — brokers differ in disk speed, in how many other partitions they host, and in how much traffic those carry. Weighting gives each partition a share proportional to capacity. Same refinement as weighted round robin in load balancing and virtual nodes sized to machine capability in consistent hashing: uniform distribution is only correct when the targets are uniform, and at fleet scale they never are."

12. Isn't exposing partition IDs to clients a leaky API?

Weak: "Yes, but it's necessary for ordering."

Strong: "It's the right call, and the reason is about who has the information. The system sees opaque payloads — it can't know these three messages belong to the same chat channel or the same database row. The semantic relationship that ordering is about exists only in the client's head. The system could invent an order, but it would be order by arrival, which is what strict ordering exists to avoid. Same conclusion as sequence numbers in the messaging queue chapter: only the producer knows what's related. The cost is that a keyed write bypasses load balancing entirely, so a skewed key gives a hot partition."

13. Why local disk instead of S3?

Weak: "It's faster."

Strong: "Messages are capped at 1 MB, so they're small — and blob stores have high per-operation cost. You'd pay an HTTP round trip, TLS, and auth to move a kilobyte, so the ceremony dominates the work, and geo-replication makes it worse. Append-only local writes go to contiguous tracks and sectors, the one pattern disks are tuned for, and contiguous regions cache extremely well. Same finding as Redis pipelining giving 5x on loopback: when the work per operation is tiny, per-operation overhead is the system. The trade is that storage is bounded by broker disk and durability becomes ours — which is why tiered storage into object stores for cold segments is where these systems are going."

14. How does replication work, and why per partition?

Weak: "Each broker has a backup broker."

Strong: "Three replicas per partition on different brokers, leader-follower, with the cluster manager electing on failure. Leadership is per partition, not per broker: broker 1 can lead partition 0 while carrying a replica of partition 1. That spreads write load — per-broker leadership would put every write for a topic on one machine — and makes failover granular, costing one partition's leadership rather than a topic. Three rather than two because two leaves no redundancy during repair, and repairs take hours at scale. And I'd spread across racks or zones, since three brokers in one rack share a failure domain."

15. How do you get exactly-once delivery?

Weak: "Acknowledge each message once."

Strong: "You don't get it from the delivery mechanism. A consumer reads, processes, and commits its offset — two independent operations, and the ordering picks the guarantee. Commit after processing → a crash in between means reprocessing → at-least-once. Commit before → a crash means the work is silently skipped → at-most-once. There's no ordering of two independent operations that gives exactly-once. You get exactly-once effects by making the handler idempotent, or by making the offset commit and the side effect atomic. Choose at-least-once and require idempotency — it also makes replay safe, since resetting an offset backwards reprocesses history."