Rollback and the Missing Gates

In one line: rollback is a stated functional requirement, and the design adopts the one deployment strategy the same chapter says has no rollback mechanism. Working out what actually survives that contradiction is the most useful thing in this chapter.

The contradiction

Three statements from the same chapter that cannot all stand

Lesson 1 (strategies): basic deployment "provides no safety net or rollback mechanism in case issues arise. It's generally unsuitable for large-scale or critical systems where reliability is essential."

Functional requirements: "Rollback: the system must support reverting to previous code versions if issues arise."

Design: "In our proposed design, we rely on a basic deployment strategy, which is flexible and straightforward to implement."

Requirements to deploy to: "thousands of machines across multiple global regions" — which is, by the chapter's own description, a large-scale system.

So the design chooses a strategy it has already declared unsuitable for the scale it is designed for, and which it says has no rollback, while rollback is a requirement.

What actually survives is narrower than the compliance table claims, and it is worth being precise about. The design does have a rollback mechanism:

"The configuration service can update the deployment version (e.g. build:1.0) to a previous one tracked by the global state service. Application servers can re-fetch and install older binaries from blob storage."

That works. Lesson 7 showed why: the declarative model makes rollback the same operation as deployment. Write the old version to the key, machines reconcile.

What it is not is a fast rollback. Compare:

Blue-green rollback: change a load balancer target        -> SECONDS
                     the old version is already running, warm

This design:         write key -> poll interval -> P2P fetch of 20 GB
                     -> install -> restart                 -> MANY MINUTES

The design has rollback-by-redeployment, which is what "no rollback mechanism" means in the strategy taxonomy. Basic deployment is not defined by an inability to run old code; it is defined by the old version not being standing by. The mechanism exists; the speed does not.

Say "the design supports rollback by redeploying the previous version, which takes as long as a deployment" rather than "the design supports rollback." The distinction is the entire point of the strategy comparison in Lesson 2.

Rollback depends on retention, and retention is never specified

You can only roll back to an artifact that still exists

Nothing in the chapter says how long artifacts are kept — not in the storage estimate, not in the schema, not in the compliance table. And rollback reads directly from that storage.

The arithmetic makes it urgent. Using the corrected figure from Lesson 3:

60 TB/day x 30 days  =  1.8 PB
60 TB/day x 365 days =   22 PB

And on the chapter's published (200×-inflated) figure, a year is 4.4 exabytes. Either way, nobody keeps everything forever, so something must be deleted — and whatever is deleted is something you can no longer roll back to.

The tension is direct:

And it is worse than a single horizon suggests, because the artifact you most need is often old. The rollback you regret not having is to the last version that was definitely fine, which may be several releases back — exactly the ambiguity Lesson 4 raised about manageRollback(deployment_id) and "the previous version."

The workable policy is tiered, and it is a good thing to have ready:

Every build                     -> keep 7-14 days   (covers normal rollback)
Every build DEPLOYED to prod    -> keep 90+ days    (covers "go back further")
Last known-good per service     -> keep INDEFINITELY (always revertible)
Tagged releases                 -> keep indefinitely (audit, compliance)

Plus the dedup from Lesson 3: with content-addressed storage, keeping an old artifact costs its delta, not 20 GB, so long retention is far cheaper than the naive figures suggest.

A retention policy is a rollback policy. Any design that claims rollback and does not state retention has claimed something it cannot guarantee.

Three of the seven stages never appear

Automated testing, staging, and automatic rollback are defined in Lesson 1 and absent from the design

Lesson 1 established that four of the seven pipeline stages move code and three exist to stop bad code. Trace the detailed design:

build -> primary blob -> replication -> regional blob -> mark ready
      -> config key -> poll -> P2P fetch -> install -> status up

Every stage that moves code is present. Two of the three that stop bad code are missing, and the third is truncated.

This is not a coincidence of exposition. It is the shape that real pipelines degrade into under delivery pressure, which is why it is worth recognizing: the stages that get skipped are always the ones that only pay off when something goes wrong.

And the omissions are self-consistent with the strategy choice. Basic deployment, no staging, no gate, no auto-rollback — these are one decision, not four. The design is optimized for the deployment succeeding.

What filling the gaps costs

Two of the three fixes need no new infrastructure at all

This is the useful part, because it is what you would say in an interview when asked to improve the design.

Staging — needs infrastructure, and less than you think. A staging environment is real cost, but it does not need to be a full-size replica. A small environment with production-shaped dependencies catches the failures staging is actually for: wrong configuration, missing migrations, incompatible dependency versions, broken service wiring. It does not catch load-related failures, which is what canary is for.

Sequenced rollout — needs no new infrastructure. Lesson 7 showed the fix: make the global key hold a per-region map rather than one version.

{ "us-west":    "build:1.1",   <- canary region
  "eu-central": "build:1.0",
  "ap-south":   "build:1.0" }

Regions already poll; they simply poll for their own entry. A promotion controller widens the rollout as health checks pass. The polling machinery exists; only a sequencer is missing. This gives you canary and staged rollout for the cost of a data-structure change.

Automatic rollback — needs a decision rule, not a mechanism. The mechanism is already there: write the old version to the key. What is missing is what decides. And the decision needs the thing Lesson 7 flagged:

"status = up"           -> liveness. Catches crashes only.
error rate vs baseline  -> catches the failures that actually need rollback
latency vs baseline     -> catches degradation

With a sequenced rollout you get the baseline for free — the regions that have not been promoted yet are the control group.

So the honest summary: the expensive missing piece is staging; sequencing and auto-rollback are mostly wiring on top of components this design already has. That is a strong thing to be able to say, because it identifies the design's actual gap as a missing controller rather than a missing architecture.

Deployment strategy shapes the architecture

The deployment strategy adopted plays a significant role in shaping its architectural decisions... a blue-green deployment requires creating duplicate environments, while a canary deployment requires mechanisms to gradually roll out changes to a subset of users. Therefore, a code deployment system should be designed to be flexible and modular.

The load balancer is the component that separates the strategies

Read the table again and notice which capability keeps recurring: control over traffic routing.

Basic and rolling deploy by replacing binaries. Blue-green and canary deploy by redirecting traffic. That is the actual dividing line, and it has a consequence for the architecture:

Binary-replacement strategies: the deployment system talks to MACHINES
Traffic-shifting strategies:   the deployment system talks to the LOAD BALANCER

This design's flow ends at "install on the machine." It never touches routing. So supporting blue-green or canary is not a matter of adding a flag — it requires the load balancer to become a controllable component of the deployment system, with an API, a state model, and a rollback path of its own.

Whether your deployment system controls traffic or only controls binaries determines which strategies you can support at all, and it is the right answer when an interviewer asks what would have to change.

The design's conclusion — "a code deployment system should be designed to be flexible and modular... to support various strategies" — is correct advice that the design does not follow. Naming the specific coupling (deployment must own routing) is what turns that advice into an architecture.

One class of failure no strategy fixes

Worth carrying into that building block, where it becomes the whole subject.

Every strategy here protects against bad code, by limiting exposure or making reversal fast. None protects against irreversible side effects:

A destructive schema migration       -> the old code cannot read the new schema
A published message consumers acted on -> cannot be unpublished
An email sent                        -> cannot be unsent
A payment captured                   -> reversal is a NEW transaction, not an undo

Rolling back the binary does not undo any of these. And the first is the most common cause of a rollback that makes things worse: revert the code, and now old code is running against a migrated database.

The discipline is the same expand-migrate-contract shape from Lesson 2:

1. Add the new column/table. Old code ignores it.       <- reversible
2. Deploy code that writes both, reads old.             <- reversible
3. Backfill. Deploy code that reads new.                <- reversible
4. Much later, drop the old column.                     <- the only irreversible step,
                                                           taken when rollback is no longer wanted

Deployment strategies protect against bad code; only design discipline protects against irreversible change. Every step must be independently reversible, or the rollback button is decoration.

Rollback is easy for code and often impossible for data, and conflating the two is the most common gap in a deployment design.

The discipline is expand and contract: make schema changes additive first so old and new code both work against the same schema, deploy the code, backfill, and only remove the old structure in a later release once rollback is no longer plausible. That decouples the two rollbacks, which is the only way the code one stays cheap.

Next: the threat model a deployment system uniquely carries.