The Build Phase

In one line: the build phase is a work-queue-and-worker-pool pattern, which is the most reusable architecture in this module. The interesting part is what makes it different from every other queue you have seen.

The flow

Developers submit code to a VCS. The CI system monitors for changes and triggers on pushes or commits.

The CI system triggers build automation by submitting code to the pub-sub system. A cluster of code-build servers retrieves the code, installs dependencies (e.g. from package.json or requirements.txt), and compiles or transpiles it into executable binaries. These artifacts are stored in primary blob storage.

The build service also executes unit tests to prevent regressions and integration tests to verify component interactions.

Why a queue

This queue is for capacity decoupling, and that is a different job from the last two you saw

The module has now used a queue three times for three different reasons, and telling them apart is the point.

Here, build jobs arrive in bursts — a team merges a release branch, CI fires twenty jobs in a minute — and the build fleet is fixed. Without a queue, either the fleet is sized for the peak and idle most of the day, or requests are rejected.

Lesson 3 quantified this: 3,000 builds/day averaged over 24 hours needs about 60 servers; the same 3,000 concentrated in an 8-hour working day needs roughly three times that. The queue is what lets you provision near the average and let the peak wait a few minutes.

Read what a queue is for before assuming its requirements. This one needs durability and throughput; it does not need strict ordering, so it can partition freely across brokers — an option that building block's queue did not have.

But order does matter in one case, and the design doesn't handle it

"Ordering barely matters" is true across projects and false within one.

Commit A merged at 10:00 -> build job A
Commit B merged at 10:02 -> build job B  (contains A plus more)

Worker 1 picks up A. Worker 2 picks up B.
B is smaller and finishes first. A finishes second and overwrites.
  -> the "latest" artifact for this project is now the OLDER commit

Nothing in the described design prevents this. Jobs are pulled by whichever worker is free, artifacts land in blob storage keyed by build, and the global state service is later pointed at a version by a human.

Two standard fixes:

Key artifacts by commit SHA, never by "latest." The schema already carries sha in all three tables, so a build produces project/abc123 rather than project/latest, and deployment names an explicit version. Then out-of-order completion is harmless — it just means artifacts appear in a different order than commits did.

Serialize per project. Partition the queue by project_name so one project's builds run in commit order, while different projects still run fully in parallel. This is the same insight as that building block's per-document queues: ordering only needs to hold within the unit that shares state.

The first is better here, because the second slows down the common case to fix a problem that explicit versioning removes entirely.

"Latest" is a dangerous name in any system where work completes out of order.

The worker pool

Build workers have three properties that make them unusually easy to operate

Worth naming, because it explains why this part of the design is short and correct.

They are stateless. A build worker holds nothing between jobs — clone the repo, install dependencies, compile, upload, forget. So workers can be added, removed, or replaced freely, and the autoscaling story is trivial.

The work is embarrassingly parallel. Two builds never interact. There is no shared mutable state, no coordination, no consistency question — the opposite of that building block in every respect.

Failure is cheap and recoverable. A build that dies costs 20 minutes of compute and is simply re-run. Compare with a payment that dies halfway, or an ordering service that dies mid-stream.

Together these mean the build fleet can be preemptible/spot capacity — the cheapest compute available, at the price of being reclaimed without notice. The heartbeat lease from Lesson 4 already handles reclamation: a worker that vanishes stops heartbeating, and the job requeues.

Stateless, independent, cheap-to-retry work is the ideal candidate for interruptible capacity, and a build fleet is the textbook case. It is also why the lease mechanism matters more here than it first appears — the design is only affordable if workers can disappear.

'Installs dependencies' hides the largest performance lever in the build phase

The description passes over it in a clause — "installs dependencies (e.g. from package.json or requirements.txt)" — and in practice this is frequently the majority of the twenty minutes.

A cold build does:

git clone            (full history, unless shallow)
dependency install   (hundreds of packages, over the network)
compile              (from scratch, no incremental state)
test                 (full suite)

Three of those four are largely identical to the previous build of the same project, and all three are cacheable:

The design's own note concedes the point without following it — "actual build times vary based on codebase size, complexity, and infrastructure (e.g. caching, build farms)" — and then the 20-minute requirement is treated as fixed.

It matters beyond speed. Caching is what makes the 20-minute budget achievable, and the 20-minute budget is what keeps batches small, which is the safety argument from Lesson 1. So the cache is a reliability mechanism wearing a performance costume.

There is a tension to state, though: a cache is shared mutable state in a system whose workers are otherwise independent. A poisoned cache entry can corrupt every subsequent build, and cache-key bugs produce the worst failure in build engineering — a build that succeeds and produces the wrong artifact. Content-addressing the cache by the hash of all inputs is what makes it safe.

The supporting services

'Distributed locks' is the phrase to interrogate

The configuration service is described as handling "synchronization, distributed locks, and reliability." What would a build worker actually lock?

Not the build itself — builds are independent, and locking them would serialize the one thing that parallelizes perfectly.

The real candidates are the shared resources around the build:

The job claim         -> only one worker may take job 42
The artifact path     -> only one writer per output location
A shared cache entry  -> one populates, others wait or proceed uncached
A limited license     -> some toolchains are seat-limited

The first is the important one, and note that it is exactly the lease from Lesson 4 — a distributed lock with a timeout is a lease, and a lease without a fencing token has the reassignment problem described there.

Any distributed lock held across a long operation is a lease, and every lease needs a fencing token, because the lock can expire while the holder is still working. Naming that connection is worth doing: the configuration service's "distributed locks" and the QueueTable's "heartbeat" are the same mechanism described twice.

Unit and integration tests run inside the build — so a test failure is a build failure, and nothing gates deployment

"The build service also executes unit tests to prevent regressions and integration tests to verify component interactions."

Putting tests inside the build is reasonable: an artifact whose tests failed should never be produced, and failing fast saves the distribution cost.

But trace what this means for the pipeline as a whole. Stage 4 of the seven-stage model — "extensive automated testing" before release — is now collapsed into stage 3. And the build's tests are, necessarily, the ones that can run in a 20-minute budget on a build worker with no production-like environment:

Runs in the build:      unit tests, integration tests against mocks/fixtures
CANNOT run in the build: end-to-end tests, load tests, tests against real
                         dependencies, anything needing a deployed environment

The second category is precisely what stages 4 and 5 exist for, and there is nowhere else in this design for them to run — no staging environment, no post-deploy validation gate.

So the design's real test posture is: whatever fits in the build budget, and then production. The validateAndTest(deployment_id, environment) API from Lesson 4 is the hook for the missing piece and is never called.

A test that runs inside the build is bounded by the build's latency budget, which is exactly why the pipeline separates them in the first place.

Next: getting a 20 GB artifact onto thousands of machines.