Offline and Online Will Disagree
In one line: offline evaluation is a simulation run on data your current system produced, so it systematically rewards agreeing with the system you are trying to replace.
The mechanism
Your logs contain interactions with items the existing policy chose to show. Nobody clicked the item ranked four-hundredth, because nobody saw it. So the log records that item as uninteresting, when in truth it records nothing about it at all.
This is exposure bias, and its consequence is precise. Offline ranking metrics measure fit to what was historically clicked. They are not designed to estimate the causal effect of changing the policy that generated the data. A model that would have surfaced something genuinely better gets penalised, because no record exists of anyone enjoying an item they were never shown.
So the disagreement is not usually a mistake. It is the expected outcome, and a candidate who presents offline numbers as proof of production impact is revealing they have not been through the loop.
Absence of a label is not a negative label
The subtler version of the same point, and worth saying separately because it changes how you train rather than only how you evaluate.
Most ranking losses treat un-clicked items as negatives. But an item can be un-clicked because the user saw it and rejected it, or because the user never saw it at all. Those are completely different signals collapsed into the same zero.
The countermeasures are worth naming: log impressions separately from results returned, so you know what was actually rendered; and weight observations by the estimated probability that the position was examined.
Before blaming the model, check the plumbing
A large share of offline-online gaps turn out to be measurement artifacts rather than model behaviour. They are cheap to check and embarrassing to miss.
| Artifact | What it looks like |
|---|---|
| Impression undercounting | Logging a recommendation as served when it never rendered on screen |
| Definition drift | Offline counts a click within the session, online within 30 seconds |
| Assignment leakage | Treatment events tagged as control, or losing their tag on a retry |
| Filtering mismatch | Offline evaluates all traffic; online excludes bots — or the reverse |
| Sample ratio mismatch | The experiment's two arms do not have the traffic split you asked for |
That last row deserves its own note, because it is the standard first check in a mature experimentation practice. If you requested a 50/50 split and observe 48/52, something is wrong with assignment or logging, and no result from that experiment can be trusted regardless of how good it looks. Mentioning a sample ratio mismatch check is a strong, specific signal.
The evaluation ladder
The practical answer is not to pick offline or online. It is to know the rungs and what each buys.
Three rungs are worth expanding, because most candidates only ever mention the first and fourth.
Counterfactual replay
Reweight the logged data to estimate how a different policy would have performed, using importance sampling: divide by the probability the old policy took the action, multiply by the probability the new one would.
It works, and it has a known weakness worth admitting. When the new policy diverges much from the old, a few log records get enormous weights and the estimate becomes extremely high-variance. Capping the weights controls the variance and reintroduces bias. There is no setting that removes both, and saying so is more credible than presenting replay as a solved problem.
It also requires that you logged the propensity — the probability the old policy assigned to what it did. If your current system is deterministic, you cannot do this at all, which is one argument for adding a small amount of deliberate randomisation to a production policy.
Interleaving
Show a single user results blended from both rankings and observe which side they engage with.
Because the comparison happens within one user rather than across two groups, it removes between-user variance — which is the dominant noise term in most engagement metrics. The result is a test that detects a difference with far less traffic than an A/B test, often by an order of magnitude.
The limitation is that it only applies where you can meaningfully blend two outputs. That is mostly ranking. You cannot interleave two pricing models.
The long-term holdback
A slice of users deliberately never given the new system, held for months.
It is how you catch harms that only appear over time — the recommender that lifts this week's engagement while narrowing what people watch until they lose interest and leave. A two-week experiment is structurally blind to that, because the damage accrues more slowly than the test runs.
It costs real money: a permanently withheld population is a permanently forgone benefit, and someone has to defend that. The usual resolution is a small percentage, held at the product level rather than per-feature.
What offline evaluation is still for
None of this makes offline evaluation useless, and saying so would overcorrect into a different wrong answer.
It is the only rung fast and cheap enough to run on every change, which makes it right for two jobs. Rejecting candidates that are clearly worse, before spending scarce experiment traffic on them. And catching regressions on a fixed set of cases you have decided must never break — which is the closest thing these systems have to a unit test.
The failure is not using offline evaluation. It is treating a filter as a verdict.
Key takeaway
Offline and online disagree for a structural reason rather than a fixable one: logged data was produced by the policy you are replacing, so offline metrics reward agreement with it, and an un-clicked item is not the same as a rejected one. Rule out instrumentation artifacts before blaming the model, then climb the ladder — replay, interleaving, A/B, and a long-term holdback for harms a short test cannot see. Offline evaluation is a filter that protects scarce experiment capacity, never proof of impact.
Next: how latency and cost budgets pick the model for you.