Artifact Distribution and Peer-to-Peer
In one line: this is the genuinely hard engineering in the chapter, and it is the only place in the module where peer-to-peer is the right answer.
The two-level hierarchy
A replication service propagates the build artifacts to regional blob storage. Assume a single build artifact can be several gigabytes in size. Downloading this artifact from regional blob storage to each machine over the network would be slow. Application servers within a region can form a peer-to-peer distribution network.
The hierarchy exists because the expensive link is the one between regions
Two levels look like extra machinery. They are there because the two hops have completely different economics.
Primary to regional — few transfers, expensive link. One copy of the artifact crosses the WAN per region. Cross-region bandwidth is the most expensive networking you buy, and latency is tens to hundreds of milliseconds.
Regional to machines — thousands of transfers, cheap link. Two thousand machines pull within a data center, where bandwidth is abundant and latency is sub-millisecond.
WITHOUT the regional tier: 2,000 machines x N regions all pull across the WAN
-> the artifact crosses the WAN thousands of times
WITH it: the artifact crosses the WAN ONCE PER REGION
-> the rest is intra-DC traffic
That is the same argument a CDN makes, and it is worth seeing that this is a CDN for binaries with a private audience. that building block put images behind a CDN for exactly this reason; here the objects are 20 GB and the consumers are servers rather than browsers, but the structure is identical.
When many consumers in one location need the same object from a distant source, put a copy in that location and fan out locally. Whether you call it a CDN, an edge cache, or a regional blob store, it is one idea.
Why client-server fails inside the region
1.07 Tbps out of one blob store — the number the chapter never computes
The bandwidth section gives 533.3 Mbps per machine and then mentions, in the same sentence, that there are 2,000 machines per region. Multiplying is the whole argument:
Per machine: 20 GB / 300 s x 8 = 533.3 Mbps
x 2,000 machines: = 1,066,600 Mbps
= ~1.07 Tbps
Over a terabit per second, from a single regional blob store, sustained for five minutes.
That is not a bottleneck to be tuned. For comparison, that building block's entire collaborative editing service — 80 million documents a day — moved about 18 Gb/s. This internal transfer, which no user ever sees, needs sixty times that in a five-minute burst.
The design makes the case qualitatively: "In client-server architectures, a central server distributes binary files to multiple clients. The server becomes a bottleneck, particularly under high load."
Correct, and enormously understated. The quantitative version turns a preference into a necessity, and it is one multiplication from numbers already on the page.
A per-unit cost is not a design input until you multiply it by the fan-out. The same defect appeared in that building block, where the twenty-collaborator broadcast was never computed, and in that building block, where the client-side suppression was described without numbers. It is the most common estimation failure in the module.
What peer-to-peer changes
In P2P architectures, the workload is distributed among multiple peers. Transfers can be made in parallel and more efficiently. P2P can take advantage of the upload and download capabilities of all the peers involved.
The mechanism is that distribution time goes from linear to logarithmic
This is the property worth being precise about, because it is what makes the difference between impossible and routine.
Client-server: the design must send N copies. Total time is bounded by N × size / source_bandwidth, and it grows linearly with the number of machines. Doubling the fleet doubles the distribution time — or requires doubling the design's link.
Peer-to-peer: each machine that finishes becomes a source. The number of available senders doubles every round:
Round 1: 1 sender -> 2 machines have it Round 2: 2 senders -> 4 Round 3: 4 senders -> 8 ... Round 11: 1,024 -> 2,048 machines have it
About eleven rounds for two thousand machines, versus two thousand sequential transfers. Time grows with log N rather than N.
And the design's own point about upload capacity is the second half: every machine in a data center has a network interface that is idle during a deployment. Client-server uses one machine's upload bandwidth; peer-to-peer uses two thousand machines' upload bandwidth. The aggregate capacity was always there.
In practice artifacts are chunked — as BitTorrent does — so a machine can serve chunk 1 while still downloading chunk 7, which removes the wait for whole-file completion and makes the doubling much finer-grained than the idealized rounds above.
When N consumers need the same large object, make the consumers into sources and the cost goes from linear to logarithmic in N. This is why Facebook, Twitter, and Netflix all built internal BitTorrent-based distribution, and it is the single most reusable idea in this chapter.
Peer-to-peer is unambiguously right here, and it is right for reasons that rarely hold
It is worth being explicit about why P2P works in this system and is a bad idea in most others, because the answer is a checklist.
| Condition | Here | Why it matters |
|---|---|---|
| Every consumer wants the identical object | ✅ The same artifact, byte for byte | Peers have nothing useful to share otherwise |
| The object is large | ✅ 20 GB | Coordination overhead must be small relative to payload |
| Consumers are in one network | ✅ One data center | Peer-to-peer across the internet is slow and asymmetric |
| Peers are trusted | ✅ Your own servers | Untrusted peers can serve corrupted data |
| Peers are simultaneously idle | ✅ Deployment is a coordinated burst | A swarm needs concurrent participants |
Change any one of these and P2P stops being attractive. This is exactly why it is not the answer for that building block's document edits (tiny, different per user), that building block's images (different per request, untrusted clients), or that building block's suggestions (small, latency-critical).
Peer-to-peer distribution requires identical large objects, a shared network, trusted peers, and simultaneous demand — a rare combination, and internal artifact distribution is its clearest instance.
The trust row deserves one qualification: peers being yours is not the same as peers being safe. A compromised machine in the swarm can serve modified bytes to two thousand others, which is why real implementations verify content hashes per chunk rather than trusting the sender. Lesson 9 returns to this.
What P2P costs
Three costs, and the third is the one that shows up in incidents
Coordination. Someone must track which peers hold which chunks — a tracker, a DHT, or the configuration service already in this design. That component becomes a dependency of every deployment.
Variable, unpredictable timing. A client-server transfer takes size / bandwidth. A swarm's completion time depends on how the topology happens to form. P2P is faster on average and less predictable, which complicates the five-minute window the estimate assumes.
Network saturation inside the data center. This is the one that surprises people. P2P does not reduce total bytes moved — it increases them, because chunks traverse multiple hops. What it reduces is bytes from the source. So a deployment can saturate the intra-rack and cross-rack fabric that production traffic is also using:
Client-server: 1.07 Tbps concentrated on ONE link (impossible)
P2P: MORE total bytes, spread across the whole fabric
-> shared with live traffic
The mitigations are the ordinary ones — rate-limit the swarm, prefer same-rack peers to keep traffic off spine links, deploy in waves rather than fleet-wide.
Peer-to-peer converts a concentrated bandwidth problem into a distributed one, which is a good trade because the concentrated version is impossible — but "distributed" means "shared with everything else."
Replication and readiness
Once regional stores are populated, the system updates a global state service to mark the binaries as ready for deployment. A key-value store tracks the target version (e.g. build:1.1).
Separating 'replicated' from 'deployable' is the right call, and it is a two-phase commit in spirit
Notice that populating the regional stores does not trigger deployment. A separate step marks the artifact ready.
That separation buys the thing you want most in a global rollout: no region starts deploying until every region can.
Phase 1 (prepare): replicate to all regional stores. Slow, retryable, invisible. Phase 2 (commit): mark ready; update the target version. Fast, atomic-ish.
Without it, a deployment could begin in Europe while replication to Asia is still running or has failed — leaving regions on different versions with no clear reason why, which is among the hardest production states to diagnose.
This is the prepare-then-commit shape, and its value is that the expensive, failure-prone work happens while nothing is committed to. The same instinct as that building block's trie: build the new thing completely, then flip one pointer.
The design does leave the failure case unspecified: what happens if replication to one region of ten fails? The reasonable answers are block the rollout (consistent, slower) or proceed and mark the region degraded (available, divergent) — and which you choose is a real decision an interviewer may push on. The prepare/commit split at least makes the choice possible; without it there is no point at which to decide.
| Hop | Copies | Mechanism | Constraint |
|---|---|---|---|
| Build → primary blob | 1 | Direct write from the build worker | None |
| Primary → regional blob | One per region | Replication service over the WAN | Expensive link — minimize crossings |
| Regional → machines | ~2,000 per region | Peer-to-peer swarm | 1.07 Tbps if client-server — impossible |
| Mark ready | 1 key | Global state service | Gate: no region deploys until all are populated |
Key takeaway
Distribution is a two-level hierarchy because the two hops have opposite economics — the WAN crossing is expensive and happens once per region, and the intra-DC fan-out is cheap and happens two thousand times. This is a CDN for binaries. Inside the region, client-server would need 1.07 Tbps from a single store, a figure the chapter never computes despite it being one multiplication from the page — and it is the whole case for peer-to-peer, which makes every receiver a sender and turns distribution time from linear into logarithmic in N. P2P is right here because of a rare five-way coincidence — identical large objects, one network, trusted peers, simultaneous demand — and it costs coordination, timing variance, and more total bytes across a fabric shared with production traffic. Finally, separating replicated from deployable is a prepare-then-commit that stops one region deploying before another can.
Next: the deployment phase and the polling loop.