Backend Service: Two Replication Models

Why this matters: this is the stateful tier — the one that actually holds messages, and therefore the one where durability is won or lost. The two models differ on a single question: is there a named owner for each queue?

Key takeaway

The backend service handles message storage, routing, and replication. The frontend queries the metadata service to find which broker is responsible for the queue, forwards the message there, and it is replicated to other nodes to improve availability and durability.

Internal versus external cluster managers

Before the models, the two kinds of manager that coordinate them:

	Internal cluster manager	External cluster manager
Scope of assignment	Manages the assignment of queues within a cluster	Manages the assignment of queues across clusters
What it knows	Knows about each and every node within a cluster	Knows about each cluster, but has no information on every host inside a cluster
Health	Listens to the heartbeat from each node	Monitors the health of each independent cluster
Failure handling	Manages host failure, instance addition, and removals from the cluster	Manages and utilizes clusters
Partitioning	Partitions a queue into several parts, each part getting a primary server	May split a queue across several clusters, so messages for the same queue are equally distributed between clusters

Model 1: Primary-secondary

Each node acts as the primary host for a specific set of queues. The primary receives requests for its assigned queues and manages data replication. The frontend consults the metadata service to identify the correct primary host for each request.

The worked example: queues 101 and 102 are assigned across four hosts (A, B, C, D). If host B is primary for queue 101, it replicates the queue data to secondary hosts such as A and C. When the frontend receives a request, it queries the metadata service — backed by the internal cluster manager — to determine the primary host. The primary serves the read request and, after receiving an acknowledgment from the consumer, deletes the message and its replicas.

The internal cluster manager maps queues to primary and secondary hosts and assists in primary host selection. This component must be reliable, scalable, and performant.

What if the internal cluster manager gets slow?

"If the internal cluster manager suddenly became slower at mapping primaries and secondaries, how might this influence the design goals of the backend service?"

It degrades every non-functional requirement at once, which is what makes it a genuine dependency rather than a helper:

Performance: the frontend consults metadata — backed by this manager — on the path of every request. Slow lookups become slow sends and receives across the board.
Availability: primary election is its job. Slower mapping means a longer window where a failed queue's primary is undetermined and the queue is unavailable.
Durability: if it is slow to notice a dead secondary and assign a replacement, the system runs under-replicated for longer — and a second failure in that window loses messages.
Scalability: adding hosts or splitting queues requires reassignment. A slow manager throttles how fast the system can grow into new capacity.

Which is exactly why the source says it must be reliable, scalable, and performant. In an interview, this is a good moment to volunteer that the cluster manager is itself a single point of failure and needs its own replication and consensus — the same conclusion the Distributed Cache chapter reached about its configuration service.

Model 2: A cluster of independent hosts

Multiple clusters of independent hosts are distributed across data centers. When the frontend receives a message, it queries the metadata service — backed by the external cluster manager — to identify the correct cluster. The message is forwarded to a random host within that cluster, which then replicates it to other hosts storing that queue.

How does a host know where to replicate?

Each host consists of a mapping between queues and hosts within a cluster, making replication easier.

Take cluster Y with hosts A, B, C, holding queues 101 and 103 on different hosts. This table is stored on each host within cluster Y. When a random host — say C — receives a message for queue 103, C replicates it to the other hosts where queue 103 is stored: A and B.

Cluster Y  (this table lives on every host in the cluster)

Queue ID | Hosts
---------|-------------------
   101   | A, C
   103   | A, B, C

A similar process applies to consumer requests: the randomly selected host handles message delivery and cleans up the message after successful processing.

This model introduces an external cluster manager to map queues to specific clusters. It handles queue management and assigns queues to appropriate clusters. The frontend routes traffic to the appropriate cluster, where nodes handle storage and delivery.

'Any host can take the write' is the whole difference

In model 1, a request for queue 101 must reach host B. In model 2, it can go to any host in the cluster, and that host handles replication itself.

The consequences follow immediately:

Availability is higher. There is no single host whose failure stalls a queue, and no election to wait through — the remaining hosts keep accepting writes.
Load spreads naturally. No single host absorbs all traffic for a hot queue, which was primary-secondary's hotspot problem.
Ordering is gone. If hosts A, B, and C can all accept messages for queue 103 concurrently, no one observes the full sequence, so no total order exists. You are firmly in best-effort territory.
Conflicts are now possible, in the way the Key-Value Store chapter described for leaderless writes.

That is the same availability-versus-consistency axis as leader-based versus leaderless replication, applied to queues. Model 1 is a leader; model 2 is leaderless.

Choose model 1 when ordering is a requirement; model 2 when availability and throughput are.

Synchronous versus asynchronous replication

"How is message replication handled in distributed queue systems?"

	Synchronous	Asynchronous
Sequence	The primary replicates the message to all relevant queues on other hosts, and only after receiving acknowledgment from secondaries does it notify the client of receipt	Once the primary receives the message it acknowledges the client, then replicates to other hosts
Gains	Messages remain consistent across all queue replicas	Lower client-visible latency
Costs	Additional communication delay, and partial or no availability while an election is in progress to promote a secondary	Replication lag and consistency issues

Based on the application's needs, we can pick one or the other.

For a queue, asynchronous replication means acknowledged messages can vanish

This deserves sharper language than "consistency issues." Under async replication, the producer is told "received" the moment the primary has the message — before any replica does.

If the primary dies in that window, the message is gone, and:

The producer believes it succeeded and will never resend.
No consumer ever sees it.
No error is raised anywhere.

Lesson 3 established that once the queue acknowledges, its copy is the only copy. Async replication means that copy briefly lives on exactly one machine. The email is never sent; the order is never fulfilled; nothing in the system knows.

For a cache, async replication is fine — the data is recoverable. For a queue, it is a durability decision with a data-loss failure mode, and it needs to be made explicitly, not defaulted into. If durability is a stated requirement, synchronous replication to at least one replica is the honest baseline, and the latency cost is the price of the requirement.

Key takeaway

Primary-secondary gives a named owner per queue — orderable, but unavailable during election. Independent clusters let any host accept a write — highly available and hotspot-free, but with no total order. Managers split hierarchically so neither one's state grows with the whole fleet. And for a queue, async replication can lose acknowledged messages silently.

Interview signal by level

Level	What a strong answer sounds like
L4	"We replicate the queue across several servers so it survives a failure."
L5	Contrasts the models: "primary-secondary gives each queue an owner that handles replication, or a cluster of independent hosts where any host takes the write and replicates from its local mapping table."
Staff+	Names what each model buys and the durability trap: "a named primary is the only way to get a total order, because someone has to observe the whole sequence — but it costs availability during election. Independent hosts are leaderless: available, no hotspot, no ordering. And I'd replicate synchronously to at least one replica, because async here means an acked message can vanish with the primary, with the producer believing it succeeded and no error anywhere."

Next: checking the design against the requirements.