Scoring a Policy You Never Ran

In one line: logged production traffic can estimate how a candidate would have performed, but only if the deployed system was randomised enough to have occasionally taken the candidate's actions — which is a logging decision made long before you need it.

The loop's bottleneck is evaluation. Shipping every candidate to real traffic is slow and risky; an offline set is fast but is not your traffic. Off-policy evaluation sits between them: real traffic, real outcomes, no deployment.

The estimator

The logged system chose action a for input x with probability p, and the outcome was r. The candidate would have chosen a with probability q. Reweight each logged record by q / p:

IPS = mean over logged records of ( q(a|x) / p(a|x) ) * r

Records the candidate would have been more likely to choose count for more; ones it would have avoided count for less. The estimate is unbiased when two conditions hold: the propensities p were actually logged at decision time, and the logging policy had non-zero probability of taking any action the candidate would take.

Variance is the practical problem

Unbiased is not the same as usable. A single record with a small p and a large q gets an enormous weight and dominates the average, so the estimate swings on one lucky trace.

The standard diagnostic is effective sample size:

ESS = (sum of weights)^2 / (sum of squared weights)

Ten thousand logged records with an ESS of 40 are forty records wearing a costume. Compute it every time and refuse to report an estimate below a floor.

The two standard remedies:

• Self-normalised IPS divides by the mean weight instead of the record count. It introduces a small bias and removes most of the variance, and it is almost always the better default.
• Doubly robust estimation combines a learned reward model with an IPS correction on its residuals. It is unbiased if either the reward model or the propensities are right, which is a genuinely useful guarantee since you rarely know which one is wrong.

Weight clipping also helps and should be reported honestly: clipping trades bias for variance, and the amount clipped is a number worth putting next to the estimate.

The support problem

The failure that no estimator repairs: the candidate wants to take actions the logged system never took. There is no data about those actions, so nothing is being estimated. This is not noise — it is absence.

It bites hardest in exactly the interesting case, a candidate that is meaningfully different from what is deployed. So off-policy evaluation is a good tool for ranking near neighbours and a bad tool for evaluating a redesign, which is worth saying plainly rather than discovering in production.

The decision that has to be made first

This is the design point of the lesson. The ability to evaluate off-policy is bought at serving time, by making the deployed policy slightly stochastic and recording the probability of the action it took. A team that skips it cannot add it retroactively — the logs simply do not contain the field, and no amount of later modelling recovers it.

The cost is a small amount of deliberate randomisation in production, which is the same investment that keeps a system's logs informative about the choices it did not make.

What it is good for, and what it is not

Worth being precise about the role this plays, because candidates tend to oversell it.

It is good for ranking a field of similar candidates against real outcomes rather than against a curated set, for catching a candidate that looks fine offline and would have behaved badly on live traffic, and for pruning a field before anyone spends experiment slots on it.

It is not a replacement for the experiment. The estimate answers what the candidate would have done on traffic the old policy shaped, not what traffic looks like once the candidate is the thing shaping it. Anything with a feedback effect — what gets shown changes what gets clicked, what gets answered changes what gets asked — is outside what the estimator models. So the honest position is that off-policy evaluation reduces how many candidates need a live test, and does not remove the last one.

Next: why the evaluation set itself wears out.