Ranking Metrics
In one line: a ranked list is not a set of classifications, because being right in position one and being right in position fifty are not the same achievement.
Why classification metrics stop working
A search engine returns twenty results. Precision would treat all twenty equally — but nobody scrolls, so a relevant item at position eighteen is worth a fraction of one at position two.
Ranking metrics exist to encode two things classification cannot: only the top matters, and order within the top matters.
The cut-off family
The simplest fix is to evaluate only the top k results.
Precision@k = relevant items in top k / k Recall@k = relevant items in top k / total relevant items
k is chosen from the product, not from statistics — 10 for a search page, 5 for a recommendation carousel, 3 for a voice assistant that reads results aloud. Saying why you chose your k is a small signal that you are reasoning from the interface.
Their limitation is that they are order-blind within the cut-off. A relevant result at position 1 and one at position 10 contribute identically to precision@10, which is exactly the distinction the user experiences most strongly.
MRR — for the one right answer
Mean reciprocal rank scores each query by the reciprocal of the position of the first relevant result, then averages.
Position 1 -> 1.00 Position 2 -> 0.50 Position 3 -> 0.33 Position 5 -> 0.20 Position 10 -> 0.10
The steep drop is deliberate: getting it first is worth twice getting it second.
MRR encodes a strong assumption — there is one right answer and everything after it is irrelevant. That fits navigational search, question answering, and "find me the customer record". It fits recommendation badly, because a carousel of ten good items is a success and MRR only ever looks at one.
MAP — for multiple relevant items
Mean average precision computes precision at each position where a relevant item appears, averages those within a query, then averages across queries.
It rewards finding all the relevant items and finding them early. It treats relevance as binary — an item is relevant or it is not — which is its main limitation.
NDCG — for graded relevance
Normalised discounted cumulative gain is the one most production ranking teams use, and it is worth understanding in three pieces because each solves a specific problem.
Gain — each result has a relevance grade, not a yes or no. Perfect, good, marginal, irrelevant.
Discounted — divide each gain by a logarithmic function of its position, so later positions contribute less.
Normalised — divide by the score of the ideal ranking, so the result sits between 0 and 1 and queries with different numbers of relevant items become comparable.
The normalisation step is the one people forget to explain and it is the one that makes averaging across queries legitimate. Without it, a query with twenty relevant documents dominates a query with one.
NDCG's cost is that it needs graded relevance labels, which usually means human annotation with a rubric. That is a real expense, and it is the reason many teams fall back to click-based binary relevance and its position bias.
Choosing between them
| Metric | Assumes | Use for |
|---|---|---|
| Precision@k | Top k matters, order within it does not | A fixed-size result page or carousel |
| Recall@k | You can enumerate all relevant items | Retrieval stages feeding a ranker |
| MRR | One right answer | Navigational search, question answering |
| MAP | Several relevant items, relevance is binary | Document retrieval with binary judgements |
| NDCG | Relevance is graded and you can label it | Production ranking where quality varies by degree |
Measure the two stages differently
This is the point that connects back to the architecture, and it is the most useful thing in the lesson.
A cascade has a retrieval stage and a ranking stage, and evaluating both with the same metric is a common and costly error.
Retrieval is judged on recall@k, where k is the candidate-set size handed to the ranker. An item retrieval drops is unrecoverable, because ranking only ever sees what it was given. Retrieval's job is to not lose things.
Ranking is judged on NDCG or precision@k over the small set it received. Its job is to order well.
The diagnostic that follows is genuinely useful and worth volunteering: if end-to-end quality is poor, measure retrieval recall first. If recall@1000 is 60%, no ranking improvement can take final quality above 60%, and every hour spent on the ranker is wasted. Teams routinely tune the ranker for months on a retrieval problem.
Key takeaway
Ranked output needs metrics that know about position: cut-off metrics are simple and order-blind, MRR assumes exactly one right answer, MAP handles several with binary relevance, and NDCG handles graded relevance and normalises so queries become comparable — at the cost of needing labelled grades. Measure the two cascade stages differently: recall@k for retrieval, NDCG or precision@k for ranking. Retrieval recall is a hard ceiling on final quality, so check it before touching the ranker.
Next: calibration, and when a score has to be an actual probability.