Position Bias and Unbiased Learning to Rank

In one line: an item at position one gets clicked far more than the same item at position ten, so training on raw clicks teaches the model to reproduce your current ranking rather than to improve it.

The contamination

Click-through rate falls steeply with position. Some of that is relevance — better items were ranked higher, so they deserved more clicks. The rest is position itself: users look at the top, and an item they never looked at cannot be clicked.

Train on raw clicks and the model cannot separate those. It learns that top-ranked items are good, which is partly a fact about relevance and partly a fact about your interface.

This is the sharpest instance of the feedback loop in the whole cascade. A model trained on raw clicks is being taught, with every example, that whatever is currently ranked highly deserves to be.

The examination hypothesis

The idea that makes correction possible, and it is worth stating precisely because it is what licenses the arithmetic.

A user clicks an item only if two things both happen: they examined it, and they found it relevant.

P(click | item, position)  =  P(examine | position)  x  P(relevant | item)

The useful assumption is that examination depends on position while relevance depends on the item. If you can estimate the first term, you can divide it out and recover the second — which is the thing you actually wanted to model.

That division is inverse propensity weighting: weight each observed click by the inverse of the probability that its position was examined.

weight = 1 / P(examine | position)

A click at position 1  ->  small weight   (it was very likely seen)
A click at position 10 ->  large weight   (it was rarely seen — this
                                           click is strong evidence)

A click deep in the list is rare and informative. Upweighting it is what stops the model from simply memorising the current order.

Estimating the propensity, and what it costs

To divide by P(examine | position) you need to know it, and this is where the honest trade-offs live.

Result randomisation. Shuffle results for a fraction of traffic and observe click rates by position with relevance held constant. Statistically clean and the standard method.

Its cost is not subtle: shuffling degrades click-through and conversion for the users who get it. You are deliberately showing people worse results in order to measure your own interface. It works and it is expensive, and any proposal to do it needs a traffic fraction and an acknowledgement of the revenue impact.

Intervention harvesting. Exploit the natural experiments already in your logs — the same item appearing at different positions across queries, or across ranker versions. Free, and it relies on those variations existing and being unconfounded, which they only partly are.

Jointly learning propensity and relevance. Estimate both from the click data at once, avoiding randomisation entirely. Attractive, and it makes stronger modelling assumptions — you are separating two things from data where they are entangled.

The last row is worth naming because it is tempting. Position-bias curves from published work are specific to a layout, a device mix and a product. Borrowing one is better than assuming no bias and worse than measuring your own, and using it should be a stated assumption rather than a silent one.

Trust bias, and the limit of the model

One more effect the examination hypothesis does not capture.

Users trust high-ranked results. Shown two identical items, one at position one and one at five, they are more likely to judge the first relevant — not because they saw it more, but because its position implied a recommendation.

That breaks the clean factorisation, because relevance now depends on position too. Correcting for examination alone leaves this residue.

The practical position: inverse propensity weighting is a large improvement over doing nothing and is not a complete correction. Saying that plainly is better than claiming debiased data is unbiased data.

What to do if none of this is affordable

A reasonable answer for a smaller system, and worth having ready.

Log position and treat it as a feature at training time, then hold it constant at serving. The model can attribute some of the click to position during training, and at inference every candidate is scored as though at the same position. It is not principled inverse propensity weighting and it captures a large part of the effect for almost no cost.

Prefer position-insensitive labels where they exist. A conversion or a purchase is far less position-driven than a click — the user has already engaged deeply, so the decision is mostly about the item.

Next: the stage that sees the list rather than the item.