Every Label You Have Is Biased

In one line: a click means the user saw it, considered it and chose it, and your training data cannot separate those three.

The problem, stated exactly

You want to model relevance: would this user value this item? What you observe is interaction: did this user interact with this item, given that your system showed it to them, in that position, in that context.

Those differ by everything the system did. And the system's choices were not random.

Four biases, four different fixes

Position bias

Items at the top get more interaction because they are at the top. The standard model is the examination hypothesis: a user interacts with an item only if they examine it and find it relevant, and examination probability falls steeply with position. So an observed click rate conflates relevance with the odds of being looked at.

The fix is inverse propensity weighting — weight each observation by the reciprocal of its examination probability. An interaction from position ten is rarer and therefore counts for more.

Which requires knowing the propensity, which requires having logged it. That is the crux: this is a logging decision, made before you have a bias problem, and no amount of later modelling recovers it.

Popularity bias

Popular items are shown more, so they accumulate more interactions, so they look better, so they are shown more. The loop is self-reinforcing and it is individually rational at every step — each impression really was the best available prediction.

Its consequence is a distribution problem rather than a per-item error: exposure concentrates on a shrinking set of winners while the long tail goes unsampled. Published work on popularity bias treats this as a multi-stakeholder issue, because the harm lands on suppliers rather than on any single user's slate.

Weighting helps a little. Only exploration actually addresses it, because only exploration generates data about items the ranking would never have surfaced.

Presentation bias

A large thumbnail earns more clicks than a small one. A video autoplaying silently earns different engagement from a static card. If a surface changes its layout, historical labels collected under the old layout are measuring something else.

Nothing corrects this after the fact. The mitigation is to log the presentation context — surface, slot geometry, media type — and treat it as a feature so the model can attribute some of the signal to it.

Trust bias

Users click top results partly because being at the top is itself a quality signal from a system they trust. That is a distinct effect from position bias — it is not about whether the item was examined, but about how it was judged once it was.

It matters because it makes the ranker's own output a self-fulfilling prophecy, and it is the hardest of the four to correct.

Getting the propensity

Three ways, in increasing order of both cost and correctness.

Randomise a slice. Shuffle results for a small percentage of traffic and measure the click rate by position directly. Cleanest estimate; costs real user experience on that slice.

Intervention harvesting. Use natural variation — the same item appearing at different positions across requests — to estimate the position effect without deliberately degrading anything. Free, and it only works where the variation already exists.

Model it jointly. Learn relevance and examination together from the same logs, with the position effect as a separate term. No experience cost, and the two effects are only separable under assumptions you cannot check.

Most production systems randomise a small slice, because the alternatives require assumptions and the slice is cheap.

What a negative actually is

The other half of the same problem: an item shown and not clicked is a weak negative, and an item never shown is not a negative at all.

Treating unshown items as negatives teaches the model that everything it did not retrieve is bad — which is precisely the closed loop, expressed as a labelling error. The unshown set is overwhelmingly items nobody has evaluated, not items anyone rejected.

The distinctions worth carrying:

• Shown, ignored — a real negative, weak, and confounded by position
• Shown, explicitly dismissed — the strongest negative available, and rare
• Never shown — unknown, and the majority of the catalogue
• Random negative — useful for training retrieval, and it says nothing about preference

Where this shows up

The audit question to ask of any recommender design, including your own: what would this system have to observe in order to be wrong about its current beliefs? If the answer is "nothing it will ever be shown", the logging is not sufficient and no model choice fixes it.

Next: what happens when a system trains on data it produced.