Second Design: Broker, Cluster Manager, Consumer Manager

In one line: four components, and the split between them is the design. Notice that the broker does data and nothing else — everything about who may read what, and which broker is alive, lives outside it.

The architecture

The broker is deliberately dumb, and that is the design

Look at what the broker does not do. It does not decide who is allowed to read a topic — that is the consumer manager. It does not know which brokers are alive — that is the cluster manager. It does not remember where any consumer has reached — the consumer manager holds offsets.

The broker stores bytes and serves reads and writes. Nothing else.

This is the same data plane versus control plane separation distributed caching noted in Redis, and it earns its keep the same way. The data path stays fast because it carries no coordination work, and the control components can be replaced, scaled, or restarted without touching the path that messages actually travel.

Contrast Lesson 4, where the message director sat on the data path doing per-message fan-out logic against a database. That is exactly the coupling this design removes.

Consumers pull — note the arrow direction

The recovered architecture labels the consumer's arrow "pull message", and that matters.

It means the consumer controls the rate, which is the backpressure argument from message-queue design: under push, an overload relocates the backlog into the consumer, which has no durability. Under pull, the backlog stays in the broker, where it is already durably stored for the retention period anyway.

Pull also composes cleanly with offsets — asking "give me from offset 4,912" is a natural pull request and an awkward push contract. Lesson 10 shows the consumer manager supports both models, but pull is the one the architecture is drawn around.

Two design considerations

Two acknowledgments, two different guarantees — keep them apart

There is a failure mode hidden in the producer ack, and it is the one that creates duplicates before a consumer is even involved:

The producer cannot distinguish "the write failed" from "the write succeeded and the ack was lost", so a retry is the only safe action and duplication is the consequence. The fix is an idempotent producer: the producer attaches a sequence number per partition, and the broker discards a record whose sequence it has already seen. Retries then become safe rather than duplicative.

That is worth pairing with the consumer-side story later, because together they are what "exactly once" actually means in practice — de-duplicated writes plus idempotent processing, not a magic delivery guarantee.

The single word "acknowledgment" covers two unrelated events, and conflating them is a common error:

• Producer-side ack = "the system has durably stored your message." This is the durability boundary. Before it, the producer must be prepared to resend; after it, the producer may forget. The messaging queue chapter's warning applies unchanged — if this ack is sent before replication completes, an acknowledged message can vanish silently.
• Consumer-side ack = "I have processed up to here." This is an offset commit, and it is what lets the consumer resume after a restart. It says nothing about durability and nothing about other subscribers.

The clean way to state it: the producer ack is about the system's obligation to the producer; the consumer ack is about the consumer's obligation to itself.

Retention as a tunable is more consequential than it looks

The seven-day default and the two extremes given — weeks for banking, immediate discard for analytics — turn retention into a first-class product decision. Three things follow:

It is a capacity decision. Storage equals ingest rate times retention, independent of how fast anyone consumes. Doubling retention doubles the disk bill with no change in traffic.

It is a compliance decision. "Banking may need weeks" is an audit requirement, and the opposite constraint exists too — regulations that require data to be gone after a period. Retention is the mechanism for both.

It is your replay window, and therefore your recovery window. Lesson 10's offsets let a consumer rewind — but only within retention. A bug discovered on day eight of a seven-day retention cannot be fixed by replay; the data is gone. That makes retention the bound on your worst-case recovery, which is a genuinely good point to raise in an interview.

Storage is relational here — and that is a considered choice

Consumer metadata, subscriptions, and retention policies go in a relational database, matching Lesson 4's reasoning: structured data with integrity guarantees.

The reason it fits: this data is low-volume, highly structured, and queried relationally — "which consumers subscribe to topic X" is a join, and subscription changes are rare. Exactly the profile the databases material identified as relational territory.

Note the deliberate split, though. Offsets do not live here. They go in a key-value store (Lesson 10), because they are high-volume, constantly overwritten, and need no relational structure. Splitting metadata by access pattern rather than putting it all in one store is the kind of choice interviewers notice.

Interview signal by level

Next: how the broker actually stores anything.