Is the Loop Working At All?

In one line: every number the loop produces was produced by the process under evaluation, so the only honest answer comes from a population the loop has never been allowed to touch.

A loop that has been running for six months reports a large cumulative gain. Some of it is real improvement, some is selection bias that survived the gates, some is the metric drifting under optimisation pressure, and some is the population changing for reasons unrelated to the system. From inside, these are indistinguishable.

The untouched holdout

A long-running holdout that stays on the original artifact answers the question directly, and it is the only mechanism that does. Population changes affect both arms, so they cancel. Metric drift affects both arms. What remains is the loop.

The costs are real and worth stating rather than hiding: the holdout population gets a worse experience over time if the loop works, the arm must be large enough to measure the effect, and someone will eventually propose deleting it because it looks like waste. That last one is the most likely way it dies. Write down at the start what it is for and what the threshold for retiring it is.

A cheaper variant when a permanent arm is impossible: reintroduce the original artifact to a small share periodically, measure the gap, and put it back. It is noisier, and it is far better than nothing.

Loop-level metrics

The artifact's quality is one thing. Whether the loop is a good machine is another, and it has its own numbers:

• Promotion rate. Very low means the gate is too tight or the proposer is weak. Very high is not a success — it usually means the gate is loose or the metric is being gamed.
• Reversal rate: promotions later rolled back. This is the loop's error rate and the single most informative number in the set.
• Confirmed-versus-selected gap, tracked over time. Widening means selection noise is growing.
• Realisation rate: the online gain divided by the offline gain that justified the promotion. A ratio well under one across many promotions means the offline evaluation is not measuring the thing.
• Time from failure cluster appearing to a promotion addressing it. This is the loop's response time, and it is the number that actually justified building it.

Evaluation usually dominates the cost, not generation — a few dozen candidates scored across a few thousand cases with a judge model is far more inference than writing the candidates was. That is worth knowing because it is also the cost that grows when you make the loop more careful, which is a real trade rather than a free improvement.

When to stop the loop

A loop is not meant to run forever, and it should have stopping conditions written down like any other bounded process. Stop when the confirmed gains fall below the cost of running it. Stop when judge-to-human agreement decays past a threshold. Stop when the reversal rate rises. Stop when the gold set stops moving while the optimisation metric keeps improving, which is the turnover signature. And stop on schedule for review, whether or not anything looks wrong.

Diminishing returns are the normal ending. Most loops deliver the majority of their value in early rounds against the largest failure clusters, then spend increasing compute on smaller gains. Recognising that and pausing is a better outcome than a loop that runs indefinitely converting inference budget into fourth-decimal improvements on a metric nobody has re-validated.

What to report, and to whom

The number that gets reported upward shapes what the loop is asked to do next, so it is worth deciding deliberately.

Report the gain measured on the untouched arm, not the cumulative sum of confirmed promotions. The second number is always larger, always available, and always partly manufactured by selection — and once it has been quoted in a review, correcting it later reads as the loop having regressed.

Alongside it, report the reversal rate and the realisation ratio. Those two say whether the loop is a reliable machine, which is a different question from whether the artifact improved, and it is the one that determines how much autonomy the loop should be given next quarter.

Next: the full design, built live.