First Design: A Queue Per Consumer

Why this matters: this design is what most candidates propose first, and it is a reasonable first answer. What separates levels is being able to say precisely why it collapses — and why the obvious repair collapses too.

Key takeaway

Producers write to topics, and consumers subscribe to topics to read messages. Since new messages are appended to the end, we can build this out of distributed messaging queues — the previous chapter's building block, reused whole.

The components

Component	Role
Topic queue	A distributed messaging queue where producers write their messages
Database	A relational database storing subscription details — which consumer subscribed to which topic, so we can deliver what they want. Relational to maintain structured data and ensure integrity
Message director	Reads messages from the topic queue, fetches subscriber details from the database, and forwards messages to the appropriate consumer queues
Consumer queue	A dedicated distributed queue for each consumer. Messages are copied here from the topic queue for consumption
Subscriber	Handles subscription requests and adds entries to the database

When a consumer subscribes, the subscription metadata is persisted to the system store. The message director reads from the topic, resolves the list of subscribed consumers, and appends the message to each consumer's queue. Consumers read messages from their assigned queues.

Note: we use failover services for the message director and subscriber to ensure reliability.

Where it breaks

A distributed messaging queue simplifies message delivery logic but introduces scalability challenges. Maintaining a separate queue for each of millions of subscribers consumes significant memory and metadata overhead. Copying the same message into each subscriber queue leads to storage amplification and redundant data replication.

Two distinct failures:

Problem	Why it's fatal
Millions of queues	Each queue carries memory and metadata overhead — its own state, its own cluster-manager entry, its own replication. Queue count scales with subscriber count
Storage amplification	The same message is copied into every subscriber's queue. One 1 MB message with a million subscribers becomes 1 TB written

Run the arithmetic — it makes the failure concrete

Take the social-media example from Lesson 1. A celebrity with 10 million followers posts one 1 KB message.

Storage written  = 1 KB x 10,000,000  = 10 GB
                   ...for ONE post.

Write operations = 10,000,000 appends across 10,000,000 queues

Every subsequent post costs another 10 GB. A hundred posts is a terabyte, for content whose original form is 100 KB.

And note what this does to the publisher's cost model. Lesson 1 said the value of pub-sub is that publishing is O(1) in subscriber count. Here it is O(N) — the message director does N appends per message. The system has taken the one property that made pub-sub worth building and thrown it away.

That is the real indictment: not that it is expensive, but that it is the wrong asymptotic class.

The obvious fix, and why it also fails

"Is there a way to avoid maintaining a separate queue for each reader?"

In messaging queues, the message disappears after the reader consumes it. So what if we add a counter for each message? The counter value decrements as a subscriber consumes the message. It does not delete the message until the counter becomes zero. Now we don't need to keep a separate queue for each reader.

One shared queue, one copy of each message, and a reference count tracking how many subscribers still need it. Storage amplification solved.

"What is the problem with the previous approach?"

Unread messages can become a bottleneck when using the conventional queue API. For example, if 9 out of 10 readers have consumed the message present at the start of the queue, then that message won't be deleted until the tenth consumer has also consumed the message — and the first nine consumers won't be able to move forward.

Shared queue with reference counting

HEAD [ m1: refcount 1 ][ m2 ][ m3 ][ m4 ] ... TAIL
       ^
       9 of 10 consumers are done with m1.
       Consumer 10 is slow / offline.

       m1 cannot be removed.
       Because a queue only serves from the HEAD,
       the other 9 consumers are STUCK behind it.

This is head-of-line blocking, and it is the most important idea in the chapter

The counter fixed storage but introduced something worse: the slowest consumer sets the pace for everyone.

The cause is not the reference count — it is the queue API itself. A queue serves from the head. If the head cannot be removed, nothing behind it can be reached. So one slow or offline subscriber freezes every other subscriber on that topic.

Sit with how badly that violates Lesson 1's promise. The entire value of pub-sub was that producers and consumers are independent — "producers are not affected by slow consumers." This design makes consumers dependent on each other, which is arguably worse, because now a single misbehaving subscriber degrades a system it does not even know it shares.

And notice the trap: both failures come from reusing the queue abstraction. Queue-per-consumer costs O(N) storage; shared-queue-with-counter costs independence. The abstraction is wrong for this problem, and no amount of patching around it helps.

What the failure tells us to build

We'll need to change the storage interface so consumers can consume data independently. The system will need to retain sufficient metadata, track which information each consumer has consumed, and delete a message once all consumers have consumed it. It resembles the reference-counting mechanism in Linux's hard links to files.

Three requirements fall out, and together they are the specification for the rest of the chapter:

One copy of each message — no per-subscriber duplication.
Independent reads — any consumer can read any position without blocking any other. This is what a queue cannot do and a log can.
Per-consumer position tracking — the system remembers where each consumer has reached.

Key takeaway

Queue-per-consumer is correct and O(N) in storage and writes — 10 GB for one 1 KB post to 10 million followers. Sharing one queue with a reference count fixes storage but creates head-of-line blocking, where the slowest subscriber freezes the rest. Both failures trace to the queue abstraction, which is why the next design replaces it with a log.

Interview signal by level

Level	What a strong answer sounds like
L4	"Give each subscriber its own queue and copy the message into all of them."
L5	Critiques it: "that's a copy per subscriber, so one message with a million subscribers is a million writes — storage amplification, plus the overhead of a million queues."
Staff+	Derives the next design from the failure: "the copy-per-consumer version is O(N) in the publisher's cost, which destroys the one property pub-sub exists for. Sharing a queue with a refcount fixes storage but gives head-of-line blocking — a queue serves from the head, so one slow subscriber freezes everyone. Both come from reusing the queue abstraction. What we actually need is an append-only log with per-consumer offsets: one copy, independent reads, deletion on retention."

Next: the architecture that does work.