Replication and Replica Placement
In one line: replication here is organized differently from every other chapter — not "three copies of a partition" but R complete groups, each able to answer any query on its own. That structure is what makes read scaling trivial.
Groups, not copies
Group 1 Group 2 Group 3 ------- ------- ------- P1 P2 P3 P4 P1 P2 P3 P4 P1 P2 P3 P4 Any one group can answer any query on its own. Add a group -> +1x read capacity.
Note: a load balancer is required to distribute queries across node groups and retry on failures.
Each group is hosted in different Availability Zones to improve performance and ensure availability if a data center fails.
Thinking in groups rather than replicas is what makes the scaling obvious
Most replication schemes describe copies per partition: "each partition has three replicas." That is true here too, but the more useful framing is the one the design uses — R complete groups, each self-sufficient.
The difference matters because a query needs every partition (Lesson 7's document partitioning fans out to all of them). So the useful unit is not "a replica of P3" but "a complete set that can serve a query end to end."
Once you see it that way, scaling reads is trivial: add a group, get another 1x of query capacity. The load balancer just has one more destination. No resharding, no data movement between existing nodes, no coordination.
This is the same shape as a stateless web tier — except each "instance" is a group of stateful nodes collectively holding the whole index. Making the replicated unit self-sufficient is what turns a stateful system into one you can scale like a stateless one.
Replication factor and placement
A replication factor of three is commonly used. Each partition is stored on three nodes. One node serves as the primary, and the other two serve as replicas. All nodes build indexes in the same order to reach a consistent state.
The worked example: four partitions, replication factor 3, two availability zones each containing two nodes. Each node acts as the primary for one partition — Node 1 in AZ₁ is primary for P1.
The three replicas of a partition are distributed across availability zones so that two copies reside in one zone and one in the other. For partition P4:
- The first replica is placed on Node 2 in AZ₂.
- The second replica is placed on Node 1 in AZ₂.
- The third replica is placed on Node 2 in AZ₁.
Each group in the illustration contains one replica from each partition (P1, P2, P3, P4).
AZ 1 AZ 2
Node 1 Node 2 Node 1 Node 2
------ ------ ------ ------
P1 PRIMARY replica replica
P2 PRIMARY replica replica
P3 replica PRIMARY replica
P4 replica replica PRIMARY
Every node: primary for one partition, replica for others.
Every partition: 3 copies, split 2-and-1 across the two AZs.
'All nodes build indexes in the same order to reach a consistent state' — this is deterministic replication
That sentence describes a specific and slightly unusual approach. Rather than one node building the index and copying it, every replica builds it independently — and they agree because they process the same documents in the same order.
This is deterministic state machine replication: identical inputs in identical order produce identical outputs, so no data transfer is needed to stay in sync.
It works because index construction is a pure function of the document set. There is no randomness, no timestamps, no external state — so the same inputs must yield the same index.
Note that Lesson 10 removes exactly this. Recomputing the index on every replica turns out to be a serious waste of CPU, and the fix is to compute once and ship the file. So this design's elegance is real and its economics are wrong — worth tracking as the chapter's second reversal.
The 2-and-1 split across two AZs is the best you can do with two zones — and it's asymmetric
With three replicas and two availability zones, you cannot spread evenly. One zone gets two copies, the other gets one.
The consequence is worth stating: losing the zone with two copies leaves you with one, so you are one failure from data loss on that partition and must re-replicate urgently. Losing the zone with one copy leaves two, which is comfortable.
So the two zones are not equally important for any given partition — but because primaries are spread across both, each zone is the "heavy" zone for some partitions and the "light" zone for others. The exposure is balanced in aggregate even though it is asymmetric per partition.
With three availability zones you would place one replica in each and lose exactly one copy per zone failure — which is why cloud providers push three-AZ deployments and why object storage's three-tier placement had a copy per failure domain.
Operating with replicas
| Operation | How it works |
|---|---|
| Indexing with replicas | Each partition is forwarded to all three replicas across both availability zones. Nodes compute the index simultaneously. If the primary node fails, indexing continues on the replicas |
| Searching with replicas | The load balancer selects one replica per partition for each query. This triples read capacity and improves availability |
'Selects one replica per partition' is a per-query load-balancing decision
Notice the granularity: the load balancer does not route a query to a group. It picks one replica per partition, independently, for each query.
That is finer-grained and better. A query can be served by P1's replica in AZ₁ and P2's replica in AZ₂ — whichever is healthiest and least loaded right now — rather than being pinned to one group whose slowest member sets the latency.
Given Lesson 7's tail-latency problem, this is the natural place to fix it: with three replicas per partition, a slow node can be avoided rather than waited on, and a hedged request to a second replica costs little.
Replication for availability and replication for latency are the same mechanism, and the per-partition choice is what lets you exploit both.
Why the primary/replica distinction is weak here
The design names a primary per partition, but for search it barely matters — any replica can answer a query, and the load balancer picks freely. That is unlike the Databases or Distributed Messaging Queue chapters, where the primary was the only node that could accept a write.
The reason is that the index is read-only on the serving path. Writes happen offline, in the indexing pipeline, and all replicas build the same thing deterministically. With no write path to serialize, there is nothing for a primary to own.
That is why "if the primary fails, indexing continues on the replicas" is a mild statement rather than a failover procedure — there is no election, no fencing, and no risk of divergence. Read-only replicas are radically simpler than read-write ones, and search gets that for free from the offline/online split.
Key takeaway
Replicate in R self-sufficient groups, each holding every partition — so adding a group adds a full 1x of read capacity with no resharding. Replication factor 3, placed 2-and-1 across two AZs, which is the best available and leaves one zone more critical per partition. Replicas stay consistent by building indexes deterministically in the same order rather than copying — elegant, and revised in the next lesson.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Replicate each partition three times across availability zones." |
| L5 | Uses the group framing: "we keep R complete groups, each holding all partitions, so any group can answer any query and adding a group adds read capacity directly." |
| Staff+ | Notes the determinism and the AZ asymmetry: "replicas stay in sync by building the index deterministically from the same documents in the same order rather than copying — index construction is a pure function, so identical inputs give identical output. With three replicas over two AZs the split is 2-and-1, so losing the heavy zone leaves one copy and needs urgent re-replication; three AZs would be cleaner. And because the index is read-only on the serving path, the primary distinction barely matters — no election, no fencing, which is far simpler than read-write replication." |
Next: what's wrong with this design.