The Eval Set Is a Depleting Resource

In one line: every decision made by looking at a holdout leaks a little information into the artifact, so a set that is queried enough times stops measuring anything, and no one is notified.

This is not contamination in the training-data sense. Nothing trained on the eval set. The leak happens through selection: each round, the candidate that scored best on this set is kept, and after enough rounds the surviving artifact is one that fits this set's particular quirks.

The mechanism

The formal treatment is adaptive data analysis, and it gives a bound worth carrying. With n examples and k adaptive queries — decisions made after seeing previous answers — the standard holdout's worst-case bias grows like sqrt(k / n).

Put numbers in it. A 10,000-example holdout and 100 adaptive decisions gives a worst-case bias around 0.1, which for an accuracy metric is ten points. That is a bound rather than a typical case, but the scaling is the point: the damage grows with the number of decisions, not with the amount of compute spent, and an automated loop makes decisions far faster than a team does.

The same literature gives the repair and its limit. Adding calibrated noise to reported answers — the reusable holdout — improves the worst case to roughly k^(1/4) / sqrt(n), letting a set support a quadratic number of queries instead of a linear one. Under a standard hardness assumption, no method does better than quadratic. There is no scheme that makes a fixed set last forever.

Running the set as a budgeted asset

Four tiers, each with a different query rate and a different assumption about how much it can be trusted. The dev set is for iteration and its number is not evidence. The selection set is resampled every round so the loop does not fit any single draw. The confirmation set is the unbiased second stage from the previous lesson. The gold set is small, human-labelled, and touched rarely.

The operational discipline is the part teams skip: count the queries. Store a counter per set, increment it on every read, and expose it. A set with no counter is a set whose remaining validity nobody can state.

Retire on the counter, not on the calendar. When a set passes its budget, sample a replacement from recent production traffic, run both for one round to measure the offset between them, and record that offset — otherwise the metric jumps at the swap and someone will attribute the jump to the loop.

The other leak

Candidate generation must not see the evaluation cases. When failure clusters are drawn from the same pool the loop is scored against, proposals get written against specific examples, and the score rises for a reason that has nothing to do with capability. Draw proposal inputs from a partition reserved for that purpose.

Where the cases come from

A set assembled once from whatever was convenient decays for a second reason: the traffic moves and the set does not. Two habits keep it honest.

Sample from recent production rather than curating, so the distribution matches, and stratify by segment so small but important slices are present in usable numbers rather than in proportion. A rare failure mode with four cases in the set cannot support a decision about it.

Keep a deliberately hard partition too — the cases the system currently fails — but score it separately. Folded into the aggregate it makes every number look worse and moves for reasons unrelated to the change being evaluated; kept separate it is the slice where a targeted candidate should show its effect.

In an interview

Most candidates say "hold out a test set" and move on. The stronger version is three sentences: a holdout is consumed by the decisions made against it, the consumption scales with the number of decisions, and an automated loop makes thousands. Then give the tiers and the counter.

Next: what happens when the loop starts consuming its own output.