Building an Eval Set

In one line: the evaluation set is the closest thing these systems have to a test suite, and a team's rate of improvement is usually bounded by its quality rather than by anyone's modelling ability.

Three sets with three jobs

Collapsing these is a common and expensive mistake, because they have incompatible requirements.

The regression set is the one teams skip and the one that pays most. Every production bug becomes a permanent test case, so the same failure cannot ship twice. It is the single cheapest evaluation practice available, and it accumulates value with no ongoing effort.

The held-out set has one rule that is easy to state and hard to keep: the more you look at it, the less it measures. Every decision made by consulting it fits your process to that specific data — an informal overfitting that no methodology detects. Look at it rarely, and treat a surprising result there as real rather than as something to iterate against.

Stratify it, or the average will lie

A set drawn from average traffic will be dominated by easy cases, because most traffic is easy. It then reports that the system is excellent while failing on everything that matters.

Deliberately over-sample the hard and the important:

Then — and this is the part most often missed — report per stratum, not only in aggregate. The whole point of stratifying is lost if you average it back together at the end. A single number over a stratified set can still hide a segment collapsing.

How big

The honest answer is that it depends on the effect you need to detect, and there is a rule of thumb worth carrying.

For a binary pass/fail metric, the uncertainty on a proportion is roughly 1 / sqrt(n):

n =   100   ->  ~10% uncertainty
n =   400   ->  ~5%
n = 1,000   ->  ~3%
n = 10,000  ->  ~1%

So a hundred examples cannot distinguish an 85% pass rate from a 90% one, and a team iterating on a hundred-item set is mostly reading noise. If you want to detect a 2% improvement, you need thousands.

That has a real consequence for judge-based evaluation: cost scales with the precision you want, and a set large enough to detect small changes may be expensive to run on every commit. The usual resolution is a tiered arrangement — a small fast set on every change, the full set nightly or before release.

Where the examples come from

Most useful sets are a mix, and the last row punches far above its volume. A case that reached a human complaint is a case someone cared about enough to report, which makes it worth more than a hundred sampled at random.

Contamination and staleness

Two ways an eval set quietly stops working, both silent.

Contamination. If your held-out examples appeared in the training data — directly, or because they came from a public benchmark a foundation model has seen — the score is memorisation rather than generalisation. Symptom: performance on your set is far better than users report. For anything built on a foundation model, assume public benchmarks are contaminated and keep a private set.

Staleness. The set was built from last year's traffic. Users have since changed how they phrase things, the product added features, the corpus grew. The set now measures performance on a distribution that no longer arrives.

The countermeasure for staleness is a refresh cadence with a specific check: sample recent production traffic, and ask whether the eval set still looks like it. When the distributions have separated, rebuild.

Next: turning a measurement into a decision.