Offline Metrics and the A/B Test Disagree

In one line: offline evaluation asks whether the model reproduces logged behaviour, the A/B test asks whether the new system changes behaviour, and those are different questions.

Five reasons, and they are structural

Not noise. Each one is a property of how the data was made.

The evaluation data came from the old policy. Your test set records what users did when the previous system chose what to show. A new model that would have shown different things is scored on how well it predicts the old system's slates — so agreement with the old policy is rewarded, and genuine improvement is invisible when it involves showing something different.

Offline metrics are pointwise; users see slates. NDCG scores an ordering of items that were actually shown. It cannot score a slate the user never saw, and it has nothing to say about diversity, redundancy or the interaction between positions.

Offline evaluation is single-shot; recommendation is repeated. A model that recommends the same excellent item every day scores well on every individual day and destroys the product over a month. The offline metric has no notion of sequence.

The horizon is wrong. Offline metrics measure immediate response. The objectives that matter — retention, satisfaction — take weeks to move, and are frequently anti-correlated with immediate engagement in the short run.

Coverage is unequal. Offline metrics are computed on items with logged interactions, which is the popular head of the catalogue. A model that is better on the long tail cannot demonstrate that on data that barely contains the long tail.

What offline evaluation is still for

Not nothing — it is a filter, not a decision.

It catches models that are outright broken. It ranks a batch of candidate models so you A/B the three most promising rather than all twenty. And it is fast, which is what makes iteration possible at all.

The correct claim is narrow and worth phrasing precisely: offline metrics are good at rejecting bad models and poor at ranking good ones. Use them as a gate on the way to an experiment, never as the decision.

Counterfactual estimation, and its limit

You can do better than naive offline evaluation by asking directly: what would have happened if the new policy had been serving?

Inverse propensity scoring is the standard tool — reweight logged outcomes by the ratio of the new policy's probability of showing an item to the old policy's. Where the two policies overlap, this gives an unbiased estimate of the new policy's performance without deploying it.

The limit is exactly the overlap. If the new policy wants to show something the old one would essentially never have shown, there is no logged data to reweight, and the variance of the estimate explodes. Doubly-robust estimators combine a reward model with the reweighting to reduce that variance, and they do not create data where none exists.

Which produces a useful conclusion: counterfactual estimation works best for small policy changes and degrades exactly where the change is most interesting. And it is another reason to log exploration impressions with their propensity — those are the observations with genuine overlap.

The experiment

A recommender A/B test has three complications a normal one does not.

Novelty and primacy. A visibly different experience produces a short-term response to the change rather than to the quality. Novelty inflates early numbers; primacy — users who preferred the old behaviour — depresses them. Both decay over one to three weeks, which sets a floor on how long the test must run regardless of how fast it reaches significance.

Interference. Treatment and control share a catalogue. If the treatment promotes an item hard, that item trends, and trending is visible to control. The arms are not independent, which understates the effect. Cluster-randomising by market or by content region reduces it, at a cost in power.

The horizon. Two weeks measures engagement. Retention takes longer, so long-running holdbacks — a small permanent slice on the old system — are how teams measure cumulative effects that no individual two-week test can see.

What to measure

Three tiers, and reporting only the first is the common failure.

Guardrails must be declared before the test. Chosen afterwards, they are whatever supports the result you got.

And a note on statistics: a recommender A/B has many metrics, and testing twenty at 5% yields a false positive by construction. Declare the primary metric in advance and treat the rest as evidence rather than as separate tests.

The failure that is invisible to all of it

Ship a change, engagement rises 3%, guardrails hold, roll it out. Over six months the catalogue narrows, the long tail stops receiving impressions, new creators leave.

No individual A/B test could have caught that. Each measured a two-week window in which the effect was fractional, and the harm is cumulative and lands on people who are not in your experiment.

The mitigations are the ones from the feedback-loop lesson: monitor distributional metrics continuously rather than per-experiment, and keep a long-running holdback that lets you compare against a system that never received any of the last year's changes.

Next: serving it inside the budget.