Requirements and Estimation

In one line: the same quantity — 3,000 deployments a day — is used to mean two different things in two adjacent sections, and the gap is a factor of 200.

Requirements

Non-functional: availability · fault tolerance ("every build must definitively report success or failure") · performance · scalability · security (encrypted credentials, all actions logged).

Two requirements are unusually well specified, and both are worth stealing

Most requirement lists in this module are vague. Two here are not.

"Every build must definitively report success or failure." This is a much stronger statement than "the system should be fault tolerant," and it rules out the failure mode that actually hurts: a build that hangs, or reports nothing, or is lost. A failed build is fine — you read the log and fix it. A build in an unknown state blocks a release and nobody knows why.

That single sentence is what forces the heartbeat mechanism in the storage schema: a worker that dies silently must be detected and its job reassigned, precisely so the outcome is never "unknown."

"Within a designated timeframe, e.g. 20 minutes." A latency budget on a build looks like a nice-to-have and it is a safety property, per Lesson 1 — long builds push teams toward large batches, and large batches are where risk concentrates.

Requirements that name a definite outcome ("success or failure, always") are stronger than requirements that name a quality ("fault tolerant"), because they can be tested.

'Environment configuration' is the requirement that causes the most production incidents

The design's own question asks why it matters, and the answer is worth being concrete about.

The same artifact runs in development, staging, and production. What differs is configuration — database connections, API endpoints, feature flags, credentials, resource limits. So:

Configuration must be separate from the artifact, or you need a different build per environment, which destroys the guarantee that what you tested is what you shipped.

Configuration must be versioned and rolled back alongside code, because a rollback that reverts the binary and leaves the new config is a combination nobody ever tested.

Configuration is where the credentials live, which makes it the highest-value data in the system.

And the failure mode this prevents is a classic: staging config pointing at production, or worse, production code pointing at a staging database. Both are configuration errors that no amount of code testing catches.

Build the artifact once and configure it per environment — the twelve-factor rule, and the reason this design has a key-value store dedicated to configuration.

Storage

Number of applications: 200 · Frequency of deployments: 3000 per day · Storage for a single build: 20 GB

200 × 3000 × 20 GB = 12 PB

The 12 PB figure and the 60-server figure cannot both be right — they differ by 200x

The arithmetic is correct. The semantics are not, and you can prove it from the chapter's own next section.

STORAGE section:  200 applications x 3000 deployments/day x 20 GB = 12 PB
                  -> treats 3000 as deployments PER APPLICATION
                  -> implies 600,000 deployments/day

SERVERS section:  "We assume 3,000 daily deployments" / 50 per server = 60 servers
                  -> treats 3000 as the TOTAL across all applications

The same number, two meanings, one section apart. Follow each reading to its conclusion:

Exactly one of the two published figures survives either way, and they are 200× apart.

The resource-estimation preamble settles it: "engineering teams deploy hundreds of services, with thousands of deployments each day." Thousands each day, across hundreds of services — 3,000 is the total. So:

Corrected storage:  3,000 x 20 GB = 60 TB/day    (not 12 PB — 200x less)
Servers:            3,000 / 50 = 60              (correct as published)

When the same number appears in two estimates, check that it means the same thing in both. This is a different failure from the DAU-as-RPS errors that ran through the earlier chapters — those used a unit wrongly and were consistently wrong. Here the unit is right in each section and the quantity's scope silently changes between them, which is harder to spot and just as consequential.

Neither figure is reduced by the two things that actually reduce it

The design notes that "real-world storage needs depend on versioning, redundancy, backups, and compression" — and omits the mechanism that dominates all of them.

Content-addressed deduplication. A 20 GB artifact is not 20 GB of new bytes. It is a base image, a runtime, dependency trees, and a small amount of application code — and between two consecutive builds of the same service, almost all of it is byte-identical.

Build N   : 20 GB
Build N+1 : 20 GB total, of which maybe 50-500 MB is new
            -> the rest is layers/blobs already stored

This is exactly what container registries and artifact stores do: hash each layer or chunk, store it once, and reference it. The effective storage cost is the delta between builds, not the size of each build.

At 3,000 builds a day the difference is enormous:

Naive:      3,000 x 20 GB          = 60 TB/day
Deduped:    3,000 x ~200 MB delta  = ~600 GB/day     -> ~100x less

When artifacts share most of their bytes, store content by hash and the estimate changes by two orders of magnitude. It is also the same idea as the P2P distribution the design uses later — both exploit the fact that the bytes already exist somewhere.

Bandwidth

A build takes 20 minutes, followed by a 5-minute transfer to a regional blob store. With 2,000 machines per region, the system must distribute 20 GB binaries to all machines within that 5-minute window.

20 GB / 5 minutes ≈ 533.3 Mbps

2,000 machines is stated, then dropped — and the number it produces is the entire justification for the design

533.3 Mbps is correct per machine. The section then stops.

But the sentence right before it says 2,000 machines per region, and that number was introduced for a reason. Multiply:

Per machine:                20 GB / 300 s x 8      =    533.3 Mbps
x 2,000 machines/region:                            = 1,066,600 Mbps
                                                    =    ~1.07 Tbps

Over one terabit per second, out of a single regional blob store, in a five-minute burst. That is more egress than most of the module's user-facing systems sustain in total, and it is generated by an internal transfer nobody sees.

Why this matters: two sections later the design introduces peer-to-peer distribution among the application servers, and justifies it qualitatively — "the server becomes a bottleneck, particularly under high load."

The quantitative justification was one multiplication away and was never made. 1.07 Tbps from one blob store is not "a bottleneck," it is an impossibility at any reasonable cost, and stating it turns the P2P decision from a preference into a necessity.

This is the same defect shape as that building block, where the twenty-collaborator fan-out that drove the whole system was never computed, and that building block, where the client-side suppression that was the largest capacity lever was described without numbers. The recurring failure is computing the per-unit cost and stopping before the fan-out.

Lesson 6 works out what P2P does to this figure.

The five-minute window is an assumption worth challenging

Where does five minutes come from? The design asserts it, and it is the denominator of the whole bandwidth calculation.

It is not a user-facing latency budget — no user is waiting. It is a pipeline pacing choice: 20 minutes to build plus 5 minutes to distribute keeps the total under half an hour, which keeps the batch small, which is the safety argument from Lesson 1.

But notice the sensitivity:

5 minutes   -> 533 Mbps/machine  -> 1.07 Tbps aggregate
15 minutes  -> 178 Mbps/machine  -> 0.36 Tbps aggregate

Tripling the distribution window cuts the bandwidth requirement by two-thirds, and costs ten minutes on a pipeline that already takes twenty for the build alone.

When a bandwidth figure comes from dividing by a self-imposed deadline, check what relaxing the deadline costs — it is often the cheapest lever available, and here nobody is waiting on the other end.

Servers

3,000 deployments / 50 per server = 60 servers

This is the first server estimate in the module whose units are actually coherent

Worth pausing on, after seven chapters of daily-active-users treated as requests per second.

deployments/day  /  (deployments/day)/server  =  servers   <- dimensionally correct

The quantity is a rate, the capacity is the same rate per server, and the quotient is servers. No unit error.

And the capacity assumption is checkable, which is rarer still. A build takes 20 minutes, so a server running builds back to back manages:

24 x 60 / 20 = 72 builds/day maximum
50 assumed   -> ~70% utilization

That is a plausible, slightly conservative figure — you would not want a build fleet at 100% utilization, because queueing delay explodes as you approach saturation and the whole point is a 20-minute budget.

The estimate is sound. The problem is that it contradicts the storage estimate, per the earlier callout — and given the requirements text, it is the storage figure that is wrong.

One thing it does hide: 60 servers is the average, not the peak. Deployments are not uniform across the day — they cluster in working hours and collapse on weekends. Sizing for the daily average leaves you queueing at 2 p.m. on a Tuesday:

3,000/day spread over ~8 working hours, not 24
  -> the working-hour rate is ~3x the daily average
  -> peak capacity closer to 180 servers, or an autoscaling build fleet

A build fleet has one of the spikiest load curves in infrastructure, which is exactly why the design queues work in a pub-sub system rather than provisioning for peak.

The building blocks

Blob storage (artifacts) · key-value store (configuration) · load balancers · pub-sub (queues build tasks when the service is busy) · monitoring (machine health).

The interesting entry is pub-sub as a work queue rather than as a notification fan-out. Its role here is load smoothing — absorbing the working-hour spike so the build fleet can be sized near the average instead of the peak.

That is a genuinely different use from that building block's operations queue, which was a serialization point whose ordering was a correctness requirement. Here order barely matters; capacity decoupling does. Read what a queue is for before assuming its requirements — the two cases need different guarantees.

The inversion worth stating: with a central artifact store, every machine you add makes the deployment slower. With peer distribution, every machine you add contributes upload capacity, so the fleet gets faster to deploy to as it grows.

Next: the APIs and the storage schema.