First Design: A Queue Per Consumer

In one line: this design is what most candidates propose first, and it is a reasonable first answer. What separates levels is being able to say precisely why it collapses — and why the obvious repair collapses too.

The components

When a consumer subscribes, the subscription metadata is persisted to the system store. The message director reads from the topic, resolves the list of subscribed consumers, and appends the message to each consumer's queue. Consumers read messages from their assigned queues.

Note: we use failover services for the message director and subscriber to ensure reliability.

Why this is a good first answer, and worth saying so

It is correct. It satisfies every functional requirement, it reuses a building block you have already designed rather than inventing storage, and each consumer genuinely gets its own independent copy with its own position.

In an interview, propose it deliberately and then critique it yourself. "Here is the design that falls out of the building block we have; here is where it breaks; here is what that tells us to build instead" is a much stronger narrative than jumping straight to partitions and offsets — because it shows you can derive the final design rather than recall it.

Where it breaks

A distributed messaging queue simplifies message delivery logic but introduces scalability challenges. Maintaining a separate queue for each of millions of subscribers consumes significant memory and metadata overhead. Copying the same message into each subscriber queue leads to storage amplification and redundant data replication.

Two distinct failures:

Run the arithmetic — it makes the failure concrete

Take the social-media example from Lesson 1. A celebrity with 10 million followers posts one 1 KB message.

Storage written  = 1 KB x 10,000,000  = 10 GB
                   ...for ONE post.

Write operations = 10,000,000 appends across 10,000,000 queues

Every subsequent post costs another 10 GB. A hundred posts is a terabyte, for content whose original form is 100 KB.

And note what this does to the publisher's cost model. Lesson 1 said the value of pub-sub is that publishing is O(1) in subscriber count. Here it is O(N) — the message director does N appends per message. The system has taken the one property that made pub-sub worth building and thrown it away.

That is the real indictment: not that it is expensive, but that it is the wrong asymptotic class.

The obvious fix, and why it also fails

"Is there a way to avoid maintaining a separate queue for each reader?"

In messaging queues, the message disappears after the reader consumes it. So what if we add a counter for each message? The counter value decrements as a subscriber consumes the message. It does not delete the message until the counter becomes zero. Now we don't need to keep a separate queue for each reader.

One shared queue, one copy of each message, and a reference count tracking how many subscribers still need it. Storage amplification solved.

"What is the problem with the previous approach?"

Unread messages can become a bottleneck when using the conventional queue API. For example, if 9 out of 10 readers have consumed the message present at the start of the queue, then that message won't be deleted until the tenth consumer has also consumed the message — and the first nine consumers won't be able to move forward.

Shared queue with reference counting

HEAD [ m1: refcount 1 ][ m2 ][ m3 ][ m4 ] ... TAIL
       ^
       9 of 10 consumers are done with m1.
       Consumer 10 is slow / offline.

       m1 cannot be removed.
       Because a queue only serves from the HEAD,
       the other 9 consumers are STUCK behind it.

This is head-of-line blocking, and it is the most important idea in the chapter

The fix is to stop letting position in a queue be the only way to address a message — which is what an offset into a shared log provides.

The counter fixed storage but introduced something worse: the slowest consumer sets the pace for everyone.

The cause is not the reference count — it is the queue API itself. A queue serves from the head. If the head cannot be removed, nothing behind it can be reached. So one slow or offline subscriber freezes every other subscriber on that topic.

Sit with how badly that violates Lesson 1's promise. The entire value of pub-sub was that producers and consumers are independent — "producers are not affected by slow consumers." This design makes consumers dependent on each other, which is arguably worse, because now a single misbehaving subscriber degrades a system it does not even know it shares.

And notice the trap: both failures come from reusing the queue abstraction. Queue-per-consumer costs O(N) storage; shared-queue-with-counter costs independence. The abstraction is wrong for this problem, and no amount of patching around it helps.

What the failure tells us to build

We'll need to change the storage interface so consumers can consume data independently. The system will need to retain sufficient metadata, track which information each consumer has consumed, and delete a message once all consumers have consumed it. It resembles the reference-counting mechanism in Linux's hard links to files.

Three requirements fall out, and together they are the specification for the rest of the chapter:

1. One copy of each message — no per-subscriber duplication.
2. Independent reads — any consumer can read any position without blocking any other. This is what a queue cannot do and a log can.
3. Per-consumer position tracking — the system remembers where each consumer has reached.

The hard-link analogy is apt, and worth extending

In a filesystem, many directory entries can point at one inode; the data is deleted only when the last link goes away. One copy, many independent references, deletion when the count hits zero.

That is exactly the shape needed here — with one crucial difference. A hard link lets each reference access the file freely, at any offset, in any order. That is the property the shared-queue-with-counter design lacked, and it is precisely what offsets into an append-only log restore.

So the fix is not a better counter. It is replacing the queue with a log — and once you have a log, the reference counting becomes almost trivial, because retention expiry handles deletion without needing to know who read what.

Interview signal by level

Next: the architecture that does work.