Cluster Manager: Replication and Failover

Why this matters: replication here happens at partition granularity, not broker granularity — and that one choice is what makes every broker useful for both leading and following.

Key takeaway

The cluster manager oversees the brokers: it maintains the broker and topics registry, manages replication via leader-follower with three replicas per partition across different brokers, and handles authorization.

Three responsibilities

Responsibility	Detail
Broker and topics registry	Maintains the list of topics per broker
Manage replication	Uses a leader-follower approach. If a leader fails, the manager elects a new one. It ensures three replicas per partition exist across different brokers
Authorization	Controls access to brokers and topics, as well as message replication across clusters

Replication at partition granularity

Each partition has one leader and two replicas, and the replicas sit on different brokers from the leader.

Per-partition leadership is why no broker is idle

Leadership is assigned per partition, not per broker. In the diagram, broker 1 leads partition 0 while simultaneously carrying a replica of partition 1, and broker 2 does the reverse.

Two things follow, and both are worth naming:

Write load spreads evenly. If leadership were per broker, one machine would take every write for a topic while its peers idled. Per-partition leadership means write traffic distributes across all brokers holding leaders.

Failover is granular. Losing broker 2 costs you the leadership of partition 1 only — partitions 0 and 2 keep serving from their own leaders, unaffected. The blast radius is one partition, not one topic.

Contrast this with the Distributed Messaging Queue chapter's primary-secondary model, where a node was primary for a set of queues and losing it stalled all of them. Splitting leadership at the partition level is a strictly finer-grained version of the same idea.

Why three replicas, specifically

Three is the standard answer across distributed systems — the Key-Value Store and Distributed Cache chapters both landed on it — and the reasoning is the same here.

Two replicas survive one failure but leave you with no redundancy during the repair window: a second failure loses data, and repairs take hours at terabyte scale. Three survives one failure and still has a spare while you rebuild.

Three also permits quorum protocols — a majority of three is two, so a write acknowledged by two survives any single loss. That gives a middle position between fully synchronous and fully asynchronous replication, which is exactly the trade the messaging queue chapter framed.

Beyond three, you pay linearly in storage for diminishing returns, which is why three is the near-universal default rather than a coincidence.

Leader failure

If a leader fails, the manager elects a new one.

What actually matters here is what the new leader is allowed to serve

Election is the visible part. The subtle part is that a promoted replica must not serve data it does not have.

If replication is asynchronous, a follower may be behind the failed leader. Promoting it means messages the old leader had acknowledged are gone — the producer was told the write succeeded, and no consumer will ever see it. This is exactly the silent-loss failure mode from the Distributed Messaging Queue chapter, and it applies unchanged.

There is a second, offset-specific hazard here. Consumers track offsets. If a promoted leader has a shorter log than the old one, an offset a consumer already holds may now point past the end or, worse, at a different message. Real systems solve this with a high-water mark — consumers may only read up to the offset replicated to all in-sync replicas, so a failover can never invalidate an offset a consumer was allowed to see.

The general rule from the cache chapter holds: a node must not serve until it can prove it is current.

Authorization, including across clusters

Controls access to brokers and topics, as well as message replication across clusters.

Cross-cluster replication is the multi-region story, mentioned in passing

"Message replication across clusters" is a single clause, but it is how these systems span regions — Kafka's MirrorMaker does exactly this.

The reason it is an authorization concern rather than just a plumbing one: replicating a topic to another cluster copies data across a trust boundary, potentially a jurisdictional one. Which topics may leave a region is a governance question before it is an engineering question.

And the same trade the cache chapter drew applies: cross-cluster replication is asynchronous — the latency makes anything else untenable — so a remote cluster is always somewhat behind. Fine for disaster recovery or feeding a regional analytics stack; not something to build strong-consistency guarantees on.

Key takeaway

Leadership is per partition, so write load spreads across brokers and a failure costs one partition rather than one topic. Three replicas on different brokers survives one failure with a spare during repair. And a promoted leader must be prevented from serving data it lacks — otherwise failover silently drops acknowledged messages and can invalidate consumer offsets.

Interview signal by level

Level	What a strong answer sounds like
L4	"Partitions are replicated so we don't lose data if a broker dies."
L5	Gets the granularity right: "three replicas per partition on different brokers, leader-follower, and the cluster manager elects a new leader if one fails."
Staff+	Names why per-partition and what failover can break: "leadership is per partition, so write load spreads and losing a broker costs one partition's leadership rather than a whole topic. On failover, a promoted replica that's behind silently drops acked messages — and it can invalidate consumer offsets, so I'd only let consumers read up to a high-water mark replicated to all in-sync replicas. And I'd spread replicas across racks or zones, since three brokers in one rack share a failure domain."

Next: the component that tracks where every consumer is.