Replication and Placement

In one line: the durability requirement from Lesson 3 — data persists until explicitly deleted — is cashed out here. Each copy defends against a different scale of failure, and knowing which is which is the whole point.

The three copies

Each copy defends a strictly larger failure domain — that's the design, not a coincidence

Read the middle column as a ladder:

• A drive dies weekly at fleet scale. The local copy handles it, at essentially zero latency cost.
• A data center loses power, cooling, or network occasionally. The second copy handles it, at intra-region latency — a millisecond or two, so it can still be synchronous.
• A region goes down rarely — natural disaster, major network partition, correlated failure. The third copy handles it, at cross-region latency, so it must be asynchronous.

Three copies in one rack would be three copies of the same failure. The value of a replica is exactly the failures it does not share with the others, which is why "how many replicas" is the less interesting question and "across which failure domains" is the real one.

This is distributed caching's warning made concrete: replicas in one data center do not survive losing that data center.

The third copy is asynchronous, and that bounds what you can promise

Lesson 12 states it: strong consistency within the storage cluster via synchronous replication, then asynchronous replication to remote data centers.

So the third copy lags. Two consequences to be honest about:

A regional disaster can lose recently written data. Blobs written in the window between the local acknowledgment and the cross-region copy are gone. The window is small, and it is not zero — this is the same acknowledged-but-not-replicated hazard message-queue design flagged.

Reads from the remote region can be stale, which is why Lesson 12 says reads are served from the primary cluster until asynchronous replication completes.

This is PACELC's else branch again, exactly as distributed caching framed it: even with no partition, synchronous cross-region writes cost too much latency, so you choose latency over consistency permanently, not only during failures.

Replication happens at chunk granularity, not blob granularity

Lesson 8's metadata table tracks replicas per chunk: d1b1, r1b1, r2b1, r3b1 for chunk 1 and a separate set for chunk 2.

That matters for repair. When a data node dies, the system re-replicates its chunks, sourced from wherever those chunks' other copies live — spread across the fleet. Repair traffic is therefore many-to-many and fast.

If replication were per-blob, a failed node's recovery would mean copying whole blobs from a small number of peer nodes, hammering them and taking far longer. Fine-grained replication makes recovery parallel, which directly shortens the window during which you are under-replicated and a second failure would lose data.

Four copies per chunk, three failure domains — keep the two counts separate

Two different numbers appear in this chapter and they are easy to conflate:

• Copies per chunk: four. Lesson 8's table is Datanode ID plus three replica IDs. Lesson 12's "four replicas per blob" counts the same set including the primary.
• Geographic placement: three tiers. Local, second data center in region, different region.

They are different axes. Four copies distributed across three failure domains means at least one domain holds two — typically two in the primary region (one per data center) plus one remote, or two locally plus one per remote tier depending on configuration.

The precise phrasing that avoids the trap: "four copies of every chunk, placed across at least three failure domains — rack, data center, and region." Say that and no follow-up can catch you out.

Why three tiers and not two or four

Two would mean choosing between surviving a data center loss and surviving a region loss — you cannot have both with one replica.

Four buys little: the failure domains above "region" are things like a provider-wide outage or a global network event, which more copies in the same provider do not address. That is a multi-cloud problem, not a replication-count problem.

Three tiers is where the marginal copy stops buying a new class of protection. That is the right way to reason about replication generally: add a copy when it defends a failure domain you do not already cover; stop when it only defends one you do.

Erasure coding: the same durability for 40% overhead instead of 200%

Replication is simple and expensive. Three copies of everything means you buy three petabytes of disk for every petabyte of data — 200% overhead — and the chapter has already argued that cost is the dominant constraint at this scale. There is a cheaper way to buy the same durability, and every real object store uses it.

Split a chunk into k data fragments, compute m parity fragments from them with Reed-Solomon, and spread all k + m across different failure domains. Any k of the k + m fragments reconstruct the original.

A 10+4 scheme tolerates more simultaneous failures than three replicas while using less than half the disk. That looks like a free lunch, and it is not — you pay somewhere else:

The two costs that actually decide it are read amplification and repair amplification. A healthy read touches k machines instead of one, so tail latency is now the slowest of ten rather than of one — bad for a hot object served to a user waiting, irrelevant for an archive. And rebuilding a single failed disk means reading k fragments for every chunk it held, so a large cluster repairing a drive generates roughly ten times the traffic replication would.

That maps cleanly onto the tiers from earlier in the chapter:

The answer that lands is not choosing one. It is: "replicate the hot tier and erasure-code the cold tier, and let the lifecycle rule that demotes a blob also be the thing that re-encodes it." Facebook's f4 and every major cloud store work exactly this way — and it is the reason archive storage can be priced an order of magnitude below hot.

Interview signal by level

Next: deletion, streaming, and caching.