Backend Service: Two Replication Models
In one line: this is the stateful tier — the one that actually holds messages, and therefore the one where durability is won or lost. The two models differ on a single question: is there a named owner for each queue?
Internal versus external cluster managers
Before the models, the two kinds of manager that coordinate them:
| Internal cluster manager | External cluster manager | |
|---|---|---|
| Scope of assignment | Manages the assignment of queues within a cluster | Manages the assignment of queues across clusters |
| What it knows | Knows about each and every node within a cluster | Knows about each cluster, but has no information on every host inside a cluster |
| Health | Listens to the heartbeat from each node | Monitors the health of each independent cluster |
| Failure handling | Manages host failure, instance addition, and removals from the cluster | Manages and utilizes clusters |
| Partitioning | Partitions a queue into several parts, each part getting a primary server | May split a queue across several clusters, so messages for the same queue are equally distributed between clusters |
Read the second row as the reason both exist
The external manager deliberately does not know what is inside a cluster. That is not a limitation — it is the whole design.
If one manager tracked every host globally, its state and heartbeat load would grow with the total fleet, and it would become the scaling bottleneck for the entire system. Splitting the responsibility means the external manager's state grows with the number of clusters (small) while each internal manager's grows with the hosts in its own cluster (bounded).
This is hierarchical coordination, and you have seen it before: DNS delegates rather than holding one global table; CDNs use tiered caches. When a coordinator's state would grow with the fleet, add a level.
Model 1: Primary-secondary
Each node acts as the primary host for a specific set of queues. The primary receives requests for its assigned queues and manages data replication. The frontend consults the metadata service to identify the correct primary host for each request.
The worked example: queues 101 and 102 are assigned across four hosts (A, B, C, D). If host B is primary for queue 101, it replicates the queue data to secondary hosts such as A and C. When the frontend receives a request, it queries the metadata service — backed by the internal cluster manager — to determine the primary host. The primary serves the read request and, after receiving an acknowledgment from the consumer, deletes the message and its replicas.
The internal cluster manager maps queues to primary and secondary hosts and assists in primary host selection. This component must be reliable, scalable, and performant.
A named owner buys ordering and costs availability
The advantage is not stated explicitly but is the reason to choose this model: because all writes for a queue go through one host, that host sees every message and can impose a total order. Lesson 4's strict ordering is only implementable when there is a single point that observes the whole sequence.
The cost is equally direct. When the primary fails, that queue is unavailable until a new primary is elected — the same failover gap the Key-Value Store and Databases chapters described. Reads and writes for queue 101 stop, even though hosts A and C hold complete copies of the data.
So: primary-secondary trades availability for orderability, and that is the right trade whenever ordering is a requirement.
What if the internal cluster manager gets slow?
"If the internal cluster manager suddenly became slower at mapping primaries and secondaries, how might this influence the design goals of the backend service?"
It degrades every non-functional requirement at once, which is what makes it a genuine dependency rather than a helper:
- Performance: the frontend consults metadata — backed by this manager — on the path of every request. Slow lookups become slow sends and receives across the board.
- Availability: primary election is its job. Slower mapping means a longer window where a failed queue's primary is undetermined and the queue is unavailable.
- Durability: if it is slow to notice a dead secondary and assign a replacement, the system runs under-replicated for longer — and a second failure in that window loses messages.
- Scalability: adding hosts or splitting queues requires reassignment. A slow manager throttles how fast the system can grow into new capacity.
Which is exactly why ** In an interview, this is a good moment to volunteer that the cluster manager is itself a single point of failure and needs its own replication and consensus — the same conclusion distributed caching reached about its configuration service.
Model 2: A cluster of independent hosts
Multiple clusters of independent hosts are distributed across data centers. When the frontend receives a message, it queries the metadata service — backed by the external cluster manager — to identify the correct cluster. The message is forwarded to a random host within that cluster, which then replicates it to other hosts storing that queue.
How does a host know where to replicate?
Each host consists of a mapping between queues and hosts within a cluster, making replication easier.
Take cluster Y with hosts A, B, C, holding queues 101 and 103 on different hosts. This table is stored on each host within cluster Y. When a random host — say C — receives a message for queue 103, C replicates it to the other hosts where queue 103 is stored: A and B.
Cluster Y (this table lives on every host in the cluster) Queue ID | Hosts ---------|------------------- 101 | A, C 103 | A, B, C
A similar process applies to consumer requests: the randomly selected host handles message delivery and cleans up the message after successful processing.
This model introduces an external cluster manager to map queues to specific clusters. It handles queue management and assigns queues to appropriate clusters. The frontend routes traffic to the appropriate cluster, where nodes handle storage and delivery.
'Any host can take the write' is the whole difference
In model 1, a request for queue 101 must reach host B. In model 2, it can go to any host in the cluster, and that host handles replication itself.
The consequences follow immediately:
- Availability is higher. There is no single host whose failure stalls a queue, and no election to wait through — the remaining hosts keep accepting writes.
- Load spreads naturally. No single host absorbs all traffic for a hot queue, which was primary-secondary's hotspot problem.
- Ordering is gone. If hosts A, B, and C can all accept messages for queue 103 concurrently, no one observes the full sequence, so no total order exists. You are firmly in best-effort territory.
- Conflicts are now possible, in the way key-value stores described for leaderless writes.
That is the same availability-versus-consistency axis as leader-based versus leaderless replication, applied to queues. Model 1 is a leader; model 2 is leaderless.
Choose model 1 when ordering is a requirement; model 2 when availability and throughput are.
Synchronous versus asynchronous replication
"How is message replication handled in distributed queue systems?"
| Synchronous | Asynchronous | |
|---|---|---|
| Sequence | The primary replicates the message to all relevant queues on other hosts, and only after receiving acknowledgment from secondaries does it notify the client of receipt | Once the primary receives the message it acknowledges the client, then replicates to other hosts |
| Gains | Messages remain consistent across all queue replicas | Lower client-visible latency |
| Costs | Additional communication delay, and partial or no availability while an election is in progress to promote a secondary | Replication lag and consistency issues |
Based on the application's needs, we can pick one or the other.
For a queue, asynchronous replication means acknowledged messages can vanish
The asymmetry that makes this worse for a queue than for most stores: the producer discards its copy on the ack. There is no upstream original to re-send from, so an acknowledged-then-lost message is lost permanently.
This deserves sharper language than "consistency issues." Under async replication, the producer is told "received" the moment the primary has the message — before any replica does.
If the primary dies in that window, the message is gone, and:
- The producer believes it succeeded and will never resend.
- No consumer ever sees it.
- No error is raised anywhere.
Lesson 3 established that once the queue acknowledges, its copy is the only copy. Async replication means that copy briefly lives on exactly one machine. The email is never sent; the order is never fulfilled; nothing in the system knows.
For a cache, async replication is fine — the data is recoverable. For a queue, it is a durability decision with a data-loss failure mode, and it needs to be made explicitly, not defaulted into. If durability is a stated requirement, synchronous replication to at least one replica is the honest baseline, and the latency cost is the price of the requirement.
Key takeaway
Primary-secondary gives a named owner per queue — orderable, but unavailable during election. Independent clusters let any host accept a write — highly available and hotspot-free, but with no total order. Managers split hierarchically so neither one's state grows with the whole fleet. And for a queue, async replication can lose acknowledged messages silently.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "We replicate the queue across several servers so it survives a failure." |
| L5 | Contrasts the models: "primary-secondary gives each queue an owner that handles replication, or a cluster of independent hosts where any host takes the write and replicates from its local mapping table." |
| Staff+ | Names what each model buys and the durability trap: "a named primary is the only way to get a total order, because someone has to observe the whole sequence — but it costs availability during election. Independent hosts are leaderless: available, no hotspot, no ordering. And I'd replicate synchronously to at least one replica, because async here means an acked message can vanish with the primary, with the producer believing it succeeded and no error anywhere." |
Next: checking the design against the requirements.