Interview Walkthrough: Design a Pub-Sub System

Why this matters: this question rewards a derivation more than a recall. The strongest structure is to propose the obvious design, break it yourself, and let the fix fall out.

Minute 0–3: Clarify

Interviewer: "Design a pub-sub system."

Weak opening: "I'd build something like Kafka — topics, partitions, offsets."

That is the destination without the reasoning, and the follow-up questions will find out whether you understand it.

Strong opening:

"Three questions first, because they move the design.

What's the fan-out? How many subscribers per topic, roughly? Ten changes nothing; ten million rules out an entire class of design, and I want to know which regime we're in.

What's the retention requirement? Messages here expire on a policy rather than on consumption, so retention sets storage capacity and it's also our replay window — which means it bounds worst-case recovery.

Do we need ordering, and at what granularity? Global ordering is enormously expensive and almost never actually required. Per-entity ordering usually is."

Interviewer: "Social-feed style. Some topics have a handful of subscribers, some have millions. Seven-day retention. Ordering matters within a single publisher's stream but not across publishers. Messages up to 1 MB."

That answer determines everything: millions of subscribers rules out copy-per-subscriber, and per-publisher ordering means partition by publisher.

Minute 3–7: The obvious design, and why it fails

"Let me start with the design that falls out of the building block we already have, then break it.

We have a distributed messaging queue. So: a topic queue producers write to, a relational database holding subscriptions, a message director that reads the topic queue, looks up subscribers, and appends the message to a per-consumer queue, and a subscriber service handling subscription requests.

That's correct. It satisfies every functional requirement. And it doesn't work at your scale.

Take a celebrity with 10 million followers posting one 1 KB message. The message director does 10 million appends across 10 million queues — 10 GB of storage for a 1 KB post. Every subsequent post costs another 10 GB.

But the deeper problem is asymptotic. The whole point of pub-sub is that publishing is O(1) in subscriber count. This design makes it O(N). It's not expensive, it's the wrong complexity class."

Interviewer: "So share one queue and reference-count the message — delete it when all subscribers have read it."

"That fixes storage and creates something worse.

A queue is served from the head. If nine of ten subscribers have consumed the head message and the tenth is slow or offline, that message can't be removed — and because there's only a head to read from, the other nine are stuck behind it. One slow subscriber freezes everyone.

That directly violates the property we're building this for: producers and consumers are supposed to be independent, and now consumers depend on each other.

Both failures come from the same source — we're reusing the queue abstraction. Copy-per-consumer costs O(N); shared-queue costs independence. There's no third option, because it's a property of the abstraction, not the implementation.

What we actually need is: one copy, independent reads at arbitrary positions, and per-consumer position tracking. That's not a queue. That's an append-only log with offsets."

Minute 7–14: The design

"Four components, and the split matters.

Broker: stores messages, serves reads and writes. Deliberately dumb — it doesn't know who's authorized, which brokers are alive, or where any consumer has reached.

Cluster manager: broker and topic registry, replication, authorization.

Consumer manager: verifies consumers, enforces retention, handles push-or-pull preference, and owns offsets.

Storage: relational, holding subscriptions, consumer metadata, and retention policies.

Both managers are control plane — nothing flows through them. That's the structural fix versus the first design, where the message director sat directly in the data path doing per-message fan-out.

Storage layout: a topic is a persistent, immutable sequence. One broker's disk I/O caps throughput, so we split topics into partitions across brokers. Each partition is a series of segments — append-only files where each message has an offset.

Segments rather than one big file because retention deletion becomes deleting the oldest segment file — O(1), no rewriting, no fragmentation.

And because records are immutable, a reader is just a cursor. Consumer A at offset 6 and consumer B at offset 2 read the same file simultaneously and cannot block each other. That's the head-of-line problem solved by the data model rather than by a protocol."

Interviewer: "How does a message get assigned to a partition?"

"Weighted round robin by default — weighted rather than plain because brokers differ in disk speed and existing load, so uniform distribution is only right when the targets are uniform.

But round robin gives no ordering at all: consecutive messages from one producer land on different partitions, and partitions are independent logs. So ordering is opt-in: the producer supplies a partition_ID in the write call, and everything with that ID lands on one log in order.

Given your requirement — order within a publisher, not across — I'd partition by publisher ID. Strict within a publisher's stream, full parallelism across publishers.

It looks odd to expose partitions to the client. The justification is that only the client knows which messages are semantically related. The system sees opaque payloads; it could only order by arrival, which isn't what anyone wants. Kafka refines this by taking a message key and hashing it, so the client says what is related rather than where to put it — I'd do that."

Interviewer: "Why not store messages in S3?"

"Because messages are capped at 1 MB, so they're small, and blob stores aren't built for constant small operations. You'd pay an HTTP round trip, TLS, and auth to move a kilobyte — the per-operation overhead dominates the work, and geo-replication makes it worse.

Local disk with append-only writes is the opposite: writes go to contiguous regions, which is the one pattern disks are tuned for, and contiguous reads cache extremely well. On spinning disks that's a three-orders-of-magnitude difference from random writes; on SSDs the mechanism changes but sequential still wins on write amplification.

The trade is real though: storage is bounded by broker disk, durability becomes our problem, and storage is coupled to compute. That's why modern systems tier — recent segments on local disk, older ones aged into object storage."

Minute 14–24: Deep dives

"A broker dies. What happens?"

"Three replicas per partition, on different brokers, leader-follower, with the cluster manager electing a new leader.

Note the granularity: leadership is per partition, not per broker. Broker 1 can lead partition 0 while carrying a replica of partition 1, and broker 2 the reverse. That spreads write load — per-broker leadership would put every write for a topic on one machine — and it means losing a broker costs one partition's leadership, not a whole topic.

Three replicas rather than two because two leaves you with no redundancy during repair, and repairs take hours at terabyte scale. Three also permits quorum, since a majority of three is two.

Two failover hazards I'd design against. If replication is async, a promoted replica may be behind, so acknowledged messages vanish silently. And offsets specifically: if the new leader's log is shorter, a consumer's stored offset may point past the end. Both are handled by a high-water mark — consumers only read up to the offset replicated to all in-sync replicas, so a failover can never invalidate an offset a consumer was allowed to see.

And I'd place the three replicas across racks or availability zones, because three brokers in one rack share a power supply and a switch."

"How do you get exactly-once delivery?"

"You don't get it from the delivery mechanism, and I'd say that directly.

A consumer reads, processes, and commits its offset. Those are two independent operations, and the ordering picks your guarantee: commit after processing gives at-least-once, because a crash in between means reprocessing. Commit before gives at-most-once, because a crash means the work is silently skipped. There's no ordering that gives exactly-once.

You get exactly-once effects two ways: make the handler idempotent so duplicates are harmless, or make the offset commit and the side effect atomic — same transaction, or an idempotency key at the business level.

For this system I'd choose at-least-once and require idempotent consumers, which is the same conclusion as the messaging queue design."

"One partition is getting hammered."

"This is the cost of keyed writes, and it's worth being honest that the system won't fix it.

Round robin guarantees even distribution. The moment a producer supplies a partition key, load balancing is bypassed — and if the key is skewed or low-cardinality, one partition takes most of the traffic while the rest idle. Your celebrity publisher is exactly this case.

Options: pick a higher-cardinality key; or split the hot key artificially — publisher:1, publisher:2, publisher:3 — with the client picking one, accepting that order is lost across the splits. For a celebrity's feed, per-message order across three sub-streams probably doesn't matter, so that's a reasonable trade.

The framing: partitioning by key trades guaranteed load balance for guaranteed ordering. You can't have both, and knowing which you're buying is the point."

"A consumer has been down for two weeks."

"Against seven-day retention, it comes back to a gap, and the dangerous part is that the failure is quiet.

Retention expiry is time-driven and doesn't consult offsets. So the consumer's stored offset now points before the start of the log. Depending on the client, its next read either errors or silently seeks to the oldest surviving message — losing a week of data with no signal.

Two defences. Alert on consumer lag measured against the retention window, not just against the head — lag in messages tells you a consumer is behind, lag against retention tells you it's about to lose data. And treat offset-before-log-start as a hard error requiring a decision, never an automatic seek-to-earliest.

More broadly: retention is the recovery window. A bug discovered on day eight can't be fixed by replay, because the data is gone. That makes retention a reliability decision, not just a storage one."

"How would you replay after a bad deploy?"

"That's the capability offsets give us almost for free. An offset is just a number in a key-value store — set it backwards and the consumer reprocesses history.

Concretely: stop the consumer, reset its offset to just before the bad deploy, deploy the fix, restart. It re-reads the same immutable messages. Nothing else on the topic is affected, because other consumers have their own offsets and the data was never destroyed by reading.

Two constraints. It only works within retention. And reprocessing means duplicate side effects, so this only works if handlers are idempotent — which we already required for at-least-once, so it composes."

If you only have five minutes

Pub-sub appears as a component in most event-driven designs:

"I'd put a pub-sub tier between the publisher and the consumers. Topics split into partitions across brokers, each partition an append-only log of immutable messages addressed by offset, with three replicas on different brokers and per-partition leadership.

Consumers pull and track their own offsets in a key-value store, so each reads the same single copy independently — no copy per subscriber, and no slow consumer blocking the others.

Ordering is per partition and opt-in via a partition key; I'd key on whatever entity needs ordering, since that key is also our parallelism unit.

Deletion is retention-driven, not consumption-driven — which makes retention our replay window and therefore our recovery window."