The Deployment Phase

In one line: the deployment mechanism is one key in a key-value store, and everything else is machines noticing it changed. That is a good design — and the interval nobody specifies determines both how fast you can ship and how fast you can recover.

The mechanism

When a deployment is triggered, the global configuration is updated with the target version:

{ build_version: build:1.1 }

Regional configuration services poll the global state. When a new version is detected, they instruct local application servers to fetch the binaries from the regional blob store. The servers download and install the update.

Changing one key to deploy globally is a declarative design, and that is worth naming

The entire deployment is: write build:1.1 to a key. Everything downstream is machines reconciling their actual state toward that desired state.

That is the declarative / reconciliation model, and it has properties an imperative "push to every machine" design does not:

The desired state is a single, inspectable fact. "What should be running?" has one authoritative answer you can read, rather than being implied by which push commands succeeded.

Machines that were down recover automatically. A server offline during the deployment comes back, polls, sees a version mismatch, and converges — with no separate catch-up mechanism. Under a push model it stays stale until someone notices.

New machines converge for free. Autoscaling adds a server, it polls, it installs the current version. No integration with the deployment system required.

Rollback is the same operation as deployment. Write build:1.0 instead of build:1.1. Same code path, same reconciliation, no special-case machinery — which is exactly the property Lesson 4 wanted from manageRollback.

This is precisely how Kubernetes works, and why. Store the desired state and let agents reconcile toward it, rather than pushing changes and hoping they all landed.

The cost is that the control plane knows what it wants, not what is — which is why monitorDeploymentProgress() and the status field exist, and why the reporting path back from machines matters as much as the command path out.

Polling

The polling interval is never specified, and it sets both deployment latency and rollback latency

The design says regional services "poll the global state" and that application servers "continuously poll the regional configuration service." No interval appears anywhere.

It is the most consequential unstated number in the chapter, because it is a two-level poll:

Total propagation delay = global->regional interval + regional->machine interval

And it cuts both ways:

The rollback column is the one that matters. The polling interval is your minimum incident duration. A five-minute poll means a bad deployment stays live for at least five minutes after you decide to revert — and that is the floor, before install and restart time.

Contrast with that building block, which used WebSockets precisely so the server could push rather than have clients poll. Deployment is far less latency-sensitive than a keystroke, so polling is defensible — but the asymmetry is worth naming:

Polling: simple, no connection state, survives client restarts,
         scales to many clients, DELAY = interval
Push:    immediate, needs persistent connections to 2,000+ machines per region

The pragmatic answer used in production is long-polling or a watch: the machine holds an open request until the value changes or a timeout fires. That gives push latency with poll semantics, and it is what etcd, Consul, and ZooKeeper all provide.

When a poll interval sets your recovery time, either shorten it or replace it with a watch.

Two-level polling is a fan-in decision, and it is the right one

Why do machines poll a regional service rather than the global one directly?

Direct: 2,000 machines x N regions all polling ONE global store
        -> 20,000+ pollers against a single strongly-consistent service

Two-level: N regional services poll the global store  (N pollers)
           2,000 machines poll their regional service (per-region load)

The global state service is the one component that must be strongly consistent — it is the single source of truth for what should be running — and strongly consistent stores are exactly the ones that do not enjoy twenty thousand pollers.

So the regional tier is a fan-in buffer, protecting a small consistent core from a large read load, while the reads that hit it are local and cheap.

Protect a strongly-consistent component by putting a caching tier in front of it, and let the tier absorb the fan-in. The same reasoning put ZooKeeper behind regional caches in the crawler and typeahead chapters, and it is the standard shape wherever a small consistent store serves a large fleet.

The full workflow

Trace what is missing from this sequence

The diagram is faithful to the design, which is the point. Read it as a checklist of what a production pipeline has and this one does not:

No test gate. Nothing between "artifact exists" and "install on production." The validateAndTest API from Lesson 4 has no place in this sequence.

No staging. selectTargetEnvironment accepts staging; no staging environment appears in the flow.

No health gate between regions. Every regional config service polls the same global key independently, so all regions deploy simultaneously. There is no "deploy region 1, watch it, then proceed" — which is the cheapest possible canary and requires no extra infrastructure, only sequencing.

No health gate within a region. All ~2,000 machines see the new version at once. This is basic deployment, per Lesson 2, with all its consequences.

No automatic rollback. Stage 7 of the pipeline promises the system "triggers a rollback to the previous stable version." Here, monitoring "notifies administrators." A human reads an alert and writes an old version to a key.

The fix for the first three is small and worth knowing, because it is nearly free given this architecture: make the global key hold a per-region schedule rather than a single version.

{ regions: { "us-west": "build:1.1",     <- canary region, first
             "eu-central": "build:1.0",  <- promoted after health check
             "ap-south":  "build:1.0" } }

Regions poll for their target. A promotion controller widens the rollout as health checks pass. The polling machinery already exists — what is missing is a sequencer in front of it, which is exactly why Lesson 2 said a deployment system should be modular enough to support several strategies.

Lesson 8 works through rollback and these gaps in full.

'Status = up' is the reporting path, and it deserves more than a status field

The workflow ends with "Once installed, the application status is set to 'up', completing the cycle."

That closes the loop — the control plane learns what actually happened — and it is the minimum viable version of it. "Up" means the process started, not that it works.

The distinction matters because the failure modes that need rollback usually pass a process-liveness check:

Process started, listening on the port          -> "up"
...but failing 30% of requests                  -> still "up"
...but 10x slower than the old version          -> still "up"
...but unable to reach its database             -> still "up"

A useful gate needs the machine to report readiness (can it serve?) and the monitoring system to compare error rate and latency against the pre-deployment baseline. That comparison is what turns "status = up" into a decision about whether to continue the rollout.

Liveness is not health, and a deployment gate built on liveness only catches crashes — which are the failures you would have noticed anyway.

What happens when a host does not come back

A deployment touches thousands of machines and some fraction will always misbehave. The design question is what the orchestrator does about it.

Deploy tasks must be idempotent, because they will be retried. Re-running a deployment on a host that already has the right version should be a no-op that confirms state rather than an action that repeats work.

Backoff with jitter, or every stuck host retries in lockstep and the orchestrator hammers the artifact store in synchronized waves at exactly the moment it is already struggling.

Quarantine rather than block. One unhealthy machine should not stop a fleet-wide rollout — set it aside, keep going, and surface it.

But count the failures. The distinction that matters: a handful of failures is bad hosts, and a large fraction is a bad build. Crossing a failure threshold should halt the rollout and roll back automatically, because continuing to push a broken version across the fleet is the failure mode this whole system exists to prevent.

Next: rollback, and the three stages the design left out.