Selection Under Noise: The Winner's Curse

In one line: picking the highest score from many noisy measurements selects for luck as well as quality, so the winner's score is biased upward by an amount that grows with the number of candidates.

This is the most common reason an offline improvement fails to appear online, and it is not the same phenomenon as offline and online metrics disagreeing. Here the metric is fine. The selection is what introduces the error.

The mechanism

Every candidate's measured score is its true score plus noise from a finite evaluation sample. Take the maximum over many candidates and you have selected, in part, for the largest positive noise draw.

For scores with roughly normal noise of size sigma, the expected maximum of N independent draws sits about sigma * sqrt(2 * ln N) above the mean.

Read the last row carefully, because it is the whole lesson. If every candidate is exactly as good as the current artifact, best-of-100 still produces a winner that appears about three noise units better. The loop will promote it, report an improvement, and the improvement is entirely an artifact of selection.

Concretely: an eval set of a few hundred cases can easily give a standard error of one accuracy point. Best-of-100 on that set then manufactures a three-point "gain" from nothing.

Why it hits automated loops harder

A human tries five variants. A loop tries hundreds, and the inflation grows with the log of that count. The loop is not doing anything wrong — it is doing more of exactly the thing that causes the bias.

It compounds across rounds too. Each round promotes an inflated winner, and the next round measures its candidates against that inflated baseline. The reported cumulative gain grows steadily while the real one flattens, which is precisely the pattern teams describe as "the loop worked for a month and then the wins stopped being real."

The fix: select, then confirm

The confirmation stage is unbiased because only one candidate is being measured — there is no maximum being taken, so there is nothing to select for. This is the shape mature prompt optimisers already use when they score candidates on minibatches and re-score the best on the full validation set, and the property that makes it work is that the second sample is not the first one.

Two rules make or break it. The confirmation data must be disjoint from the selection data, or the bias comes straight back. And the promotion decision must use the confirmed number, not the selection number — reporting the selection score is where most of the exaggeration in these systems comes from.

The diagnostic it hands you for free

The gap between the selection score and the confirmation score is a measurement of your evaluation's noise under selection. Track it per round.

A gap consistently near zero means the eval set is large enough for the field size. A large gap means either the eval set is too small or the candidate field is too big — and both are adjustable. A gap that grows across rounds usually means candidates have become more similar to each other, so more of the ranking is noise.

Related traps

Sequential testing without correction has the same shape in time rather than across candidates: checking a running experiment repeatedly and stopping when it looks significant selects for the moment the noise was favourable.

And re-using one eval set across many rounds is the same bias accumulating slowly — which is its own subject.

Next: scoring a change without shipping it.