Learning to Rank

In one line: ranking is not regression, because the loss you care about depends on the order of the whole list and every position is weighted differently.

Why pointwise is the wrong shape

The obvious approach: label each query-document pair with a relevance grade, train a regressor, sort by predicted score.

It works, and it optimises the wrong thing. Squared error treats a mistake at position one and a mistake at position ninety identically — but users see the first three results and almost never the ninetieth. The loss has no notion of position, while the metric you actually report is dominated by it.

There is a second problem. Relevance is relative to the query. A document scoring 0.7 might be the best result for a hard query and the worst for an easy one. Pointwise training pools all queries together and asks the model to produce absolute scores that are comparable across queries — which is a harder problem than the one you need solved, since you only ever sort within a query.

Pairwise

Reframe it: given two documents for the same query, which should rank higher? That is binary classification on pairs, and it matches what ranking actually is — an ordering, not a set of absolute values.

RankNet is the canonical version: model the probability that document i outranks document j as a sigmoid of their score difference, and train with cross-entropy. Scores become comparable within a query and meaningless across queries, which is exactly right.

Pairwise fixes the comparability problem and not the position problem. Swapping the documents at ranks one and two matters enormously; swapping ranks fifty and fifty-one does not. A plain pairwise loss weights both the same.

The lambda trick

This is the idea worth being able to explain, because it is elegant and it is what LambdaMART is built on.

The metric you want to optimise — NDCG — is not differentiable. It depends on ranks, and ranks are discrete, so you cannot take a gradient of it.

The insight: you do not need the metric's gradient. You need a gradient for each pair, and you can weight that gradient by how much NDCG would change if you swapped that pair. A swap at positions one and two changes NDCG a lot, so its gradient is scaled up. A swap at fifty and fifty-one barely moves NDCG, so its gradient nearly vanishes.

lambda_ij  =  (pairwise gradient)  x  |change in NDCG from swapping i and j|

The model never computes NDCG's derivative. It computes a pairwise gradient and multiplies by a listwise quantity, which injects the metric's position weighting into a pairwise framework.

LambdaMART is that lambda gradient inside gradient-boosted trees. It remains a very strong baseline on tabular ranking features — and the reason is not mysterious: search ranking features are heterogeneous tabular data, which is what boosted trees are best at.

Listwise

Treat the whole result list as one training example and optimise a loss defined over it — ListNet, ListMLE, or a smoothed NDCG surrogate.

Most principled and most expensive. The loss is over permutations, and training is slower and less stable. In practice LambdaMART's pairwise-with-listwise-weighting occupies the pragmatic middle, which is why it has stayed dominant for so long.

Features are where the wins are

The model matters less than what you feed it, and the feature groups are worth knowing because "what features?" is a guaranteed follow-up.

The third group is where ranking gets its power, and it is the group that cannot be precomputed — a query-document feature exists only for a pair, so it is computed at request time for the surviving candidates. That is the reason the funnel exists here as it does in recommendation: cross features are unaffordable at retrieval scale and decisive at ranking scale.

Labels: judgements or clicks

Two sources, with opposite properties.

Human judgements are unbiased, expensive, sparse, and reflect a rater's opinion rather than a user's satisfaction. They give a stable benchmark that does not move when your system does.

Clicks are abundant, free, and biased by position, presentation and by whatever your current ranker chose to show. Training on raw clicks teaches the model to reproduce the current ranking.

Production systems use both: judgements for a stable evaluation set and often for a base model, clicks — debiased — for volume. The unbiased-LambdaMART line of work is precisely about learning the position bias jointly with the ranker so click data can be used without baking in the current system's ordering.

When not to learn a ranking

The judgement worth showing, and the same one as in the last chapter.

Learning to rank needs labelled data, whether judgements or enough clicks. Below that, a hand-tuned combination of BM25 across weighted fields plus a few quality signals is better, cheaper, debuggable, and it works on day one. An enterprise search with a thousand queries a day does not have the signal to train a ranker or the traffic to evaluate one.

Ship the tuned baseline, collect judgements and clicks, and learn a ranker when there is something to learn from.

Next: the query distribution, and why the head and the tail need different systems.