Second Design: Broker, Cluster Manager, Consumer Manager

Why this matters: four components, and the split between them is the design. Notice that the broker does data and nothing else — everything about who may read what, and which broker is alive, lives outside it.

Key takeaway

Four components: the broker (storage and read/write requests), the cluster manager (broker health), storage (consumer and subscription metadata), and the consumer manager (authorization and access).

The architecture

Component	Responsibility
Broker	Handles message storage and read/write requests
Cluster manager	Monitors broker health and manages the cluster, notifying the system of any failures
Storage	A relational database storing consumer metadata, subscription info, and retention policies
Consumer manager	Manages consumer authorization and access

The broker is deliberately dumb, and that is the design

Look at what the broker does not do. It does not decide who is allowed to read a topic — that is the consumer manager. It does not know which brokers are alive — that is the cluster manager. It does not remember where any consumer has reached — the consumer manager holds offsets.

The broker stores bytes and serves reads and writes. Nothing else.

This is the same data plane versus control plane separation the Distributed Cache chapter noted in Redis, and it earns its keep the same way. The data path stays fast because it carries no coordination work, and the control components can be replaced, scaled, or restarted without touching the path that messages actually travel.

Contrast Lesson 4, where the message director sat on the data path doing per-message fan-out logic against a database. That is exactly the coupling this design removes.

Two design considerations

Consideration	Detail
Acknowledgment	Confirms successful message storage (producer side) or consumption (consumer side)
Retention time	A configurable message lifespan. The default is seven days, but requirements vary — banking applications may need weeks, while analytical applications may discard data immediately after use

Two acknowledgments, two different guarantees — keep them apart

The single word "acknowledgment" covers two unrelated events, and conflating them is a common error:

Producer-side ack = "the system has durably stored your message." This is the durability boundary. Before it, the producer must be prepared to resend; after it, the producer may forget. The messaging queue chapter's warning applies unchanged — if this ack is sent before replication completes, an acknowledged message can vanish silently.
Consumer-side ack = "I have processed up to here." This is an offset commit, and it is what lets the consumer resume after a restart. It says nothing about durability and nothing about other subscribers.

The clean way to state it: the producer ack is about the system's obligation to the producer; the consumer ack is about the consumer's obligation to itself.

Retention as a tunable is more consequential than it looks

The seven-day default and the two extremes given — weeks for banking, immediate discard for analytics — turn retention into a first-class product decision. Three things follow:

It is a capacity decision. Storage equals ingest rate times retention, independent of how fast anyone consumes. Doubling retention doubles the disk bill with no change in traffic.

It is a compliance decision. "Banking may need weeks" is an audit requirement, and the opposite constraint exists too — regulations that require data to be gone after a period. Retention is the mechanism for both.

It is your replay window, and therefore your recovery window. Lesson 10's offsets let a consumer rewind — but only within retention. A bug discovered on day eight of a seven-day retention cannot be fixed by replay; the data is gone. That makes retention the bound on your worst-case recovery, which is a genuinely good point to raise in an interview.

Key takeaway

The broker is intentionally dumb — storage and serving only. Cluster manager handles broker health, consumer manager handles who may read and where they are, storage holds subscriptions and policies. Retention is a capacity, compliance, and recovery-window decision all at once.

Interview signal by level

Level	What a strong answer sounds like
L4	"Brokers store the messages and consumers read from them."
L5	Separates the tiers: "the broker just stores and serves; a cluster manager watches broker health and a consumer manager handles authorization and offsets."
Staff+	Names the plane split and the retention consequence: "keeping the broker dumb is a data-plane/control-plane separation — the data path carries no coordination, so control components can restart without touching message flow. And retention isn't just a storage knob: it's the replay window, so it bounds worst-case recovery. A bug found after the retention period can't be fixed by replaying."

Next: how the broker actually stores anything.

Second Design: Broker, Cluster Manager, Consumer Manager

The architecture

Two design considerations

Interview signal by level

Enjoying the preview?