Promotion, Rollback and Blast Radius

In one line: an automated promotion is only as safe as its rollback, so design the reversal first and let it determine how much traffic a new artifact is ever allowed to touch.

Everything so far produces a candidate with a confirmed estimate. The remaining question is the one an interviewer will push hardest on, because it is where the system stops being an experiment: what makes it acceptable for code to change production without asking anyone.

The staged path

Shadow deserves more credit than it gets in these designs. It is the only stage that costs nothing in user harm, and it catches the entire class of failures that are about the artifact being broken rather than worse — malformed output, a tool call that no longer parses, a latency regression, a cost blow-up. Those are common in generated artifacts and they are cheap to catch here.

What the promotion criteria have to include

A single "is it better" test is not enough. The gate needs four kinds of check, and they fail differently:

• The primary metric improved by more than its noise, measured on confirmation data the selection never saw.
• No guardrail regressed past its floor: cost, latency, safety violations, refusal rate, output length.
• No segment regressed past its floor, so an aggregate gain cannot be paid for by a minority slice.
• The change is within the pressure budget — bounded distance from the current artifact, and within the allowed number of automatic promotions since the last human review.

The last one is the one that makes a system self-improving rather than self-modifying without limit. A loop that may promote indefinitely without review will, given enough rounds, arrive somewhere no one chose.

Rollback is the load-bearing property

That test is the cleanest available line between what a loop owns and what it recommends. It also settles the artifact-ladder question from earlier without needing a separate argument: prompts and configuration pass it easily, a fine-tuned checkpoint passes it if checkpoints are retained and loadable, and anything requiring a schema migration fails it outright.

Rollback needs to be automatic as well as possible. Define trigger conditions in advance — a guardrail breach, an error-rate step change, a metric falling below the pre-promotion baseline — and let the system revert on them without waiting for a human to interpret a graph. Then alert. The reverse order is how a bad artifact stays live for a shift change.

Versioning, or the system is not debuggable

A system that rewrites itself overnight is impossible to investigate unless every response can be traced to the exact artifact that produced it.

Stamp a version identifier on every response and store it in the trace. Keep artifacts immutable and content-addressed, so a version is a hash rather than a mutable pointer. Record, for each promotion, what changed, which candidate it was, what the confirmation numbers were, and which evaluation sets were used. Keep the previous version loadable at all times.

The everyday value of this is answering "why did this response look like that" for something that happened three promotions ago. The occasional value is much higher: when a slow regression is finally noticed, the version history is the only way to bisect it.

Where the human belongs

Not inside the edit, and not absent. Reviewing the diff of what changed on a fixed cadence, holding the authority to promote anything that fails the rollback test, and owning the metric definition and the gold set. That is a small enough job to actually get done, which is the test of whether an oversight design is real.

Next: whether the loop is working at all.