Evaluating a Cascade
In one line: four stages and one output means a single quality metric tells you something is wrong and never what.
A metric per boundary
| Stage | Metric | A low number means |
|---|---|---|
| Candidate generation | Recall@k of finally-shown items | A hard ceiling on everything after it |
| Pre-ranking | Agreement with the ranker's top-k | Discarding what the ranker wanted |
| Ranking | NDCG or precision@k over what it received | Right items, wrong order |
| Re-ranking | Diversity, quota compliance, constraint violations | Individually good, collectively bad |
| End to end | The product metric — the only one that decides | Something upstream; the number cannot say what |
The order of investigation follows the compounding ceiling: check the earliest stage first, because a loss there cannot be recovered by any improvement downstream.
The diagnostic sequence. Recall at candidate generation. Then pre-ranker agreement. Then ranking quality on what it actually received. Then set-level properties. The same discipline that orders a retrieval-augmented pipeline's diagnosis, applied here.
What "recall" means at each stage
A subtlety worth being precise about, because the word does different work at different boundaries.
At candidate generation, recall is measured against the items that would have been good — which you do not know. The workable proxy is: of the items the user eventually engaged with, what fraction did candidate generation return? Biased, since it only counts items that were shown, and it is the number available and it moves in the right direction.
At pre-ranking, recall is measured against the ranker, not against ground truth. Score the full candidate set with the ranker offline, and ask what fraction of its top-k the pre-ranker forwarded. This one is clean, because the reference is a model you control and can run offline over anything.
That difference matters: the pre-ranker's metric is exactly measurable and the retriever's is not. Which is a reason to trust one number more than the other when they disagree.
The offline metrics that lie
Everything above is offline, and the evaluation chapter established why offline ranking numbers overstate: judgements exist only for items the current system showed, so a model that surfaces something genuinely better is scored as if it surfaced something irrelevant.
That applies with full force here, and one instance is worth calling out specifically. A change to candidate generation is the hardest thing to evaluate offline, because its entire value is retrieving items the old system never surfaced — and those have no labels by construction. Offline recall will show little or no gain from a change whose whole point is finding what you were missing.
The practical consequence: for retrieval changes, offline evaluation is a safety check — did anything obviously break — and the verdict has to come from interleaving or an experiment.
Counterfactual and interleaving
Interleaving deserves a specific mention here because ranking is exactly the case it was built for.
Blend two rankings for the same user and observe which side they engage with. Because the comparison is within a user rather than across two groups, it removes between-user variance — the dominant noise term — and detects a difference on far less traffic than an A/B test.
Its constraint is that it needs two rankings over a comparable candidate set. So it works cleanly for a ranker change and awkwardly for a candidate generation change, where the two systems produce different sets and interleaving them compares things that are not alternatives.
The metrics that catch slow harm
Quality metrics measure this request. Some failures only appear across many.
Catalogue coverage. What fraction of the catalogue was shown to anyone this week? A falling number is popularity collapse in progress — the feedback loop, made measurable.
Gini or concentration of impressions. How unevenly are impressions distributed across items? Rising concentration means the system is narrowing.
New-item impression share. Are new items getting shown at all? This is the exploration quota's actual outcome, as opposed to its configured value.
Per-user diversity over time. Is any individual's feed narrowing? Filter bubbles are per-user popularity collapse.
What to report
A ranking system's dashboard should carry three groups, and the third is the one usually missing:
Per-stage health — recall, agreement, NDCG, constraint violations.
Product outcome — the primary metric with its guardrails.
Ecosystem health — coverage, concentration, new-item share, per-user diversity.
The third group is what tells you the system is quietly eating its own future, and it needs to be on the same page as the first two or nobody will look at it.
Key takeaway
Measure at every stage boundary and investigate earliest-first, because the ceilings compound and an early loss cannot be recovered. Note that the metrics differ in trustworthiness: pre-ranker agreement is exactly measurable against a model you control, while candidate recall can only be measured against items that were shown and therefore flatters. Retrieval changes are the hardest to evaluate offline, since their value is precisely the items with no labels — so the verdict comes from interleaving or an experiment. And report ecosystem health alongside quality, because coverage and concentration degrade for months while engagement still looks fine.
Next: the walkthrough, where the whole cascade gets designed live.