Why More Than One Stage

In one line: the two-stage story is the simplification, real systems run three or four, and the extra stage exists for a reason worth being able to state.

The arithmetic, again

A ranking request has a latency budget of a few hundred milliseconds at p99, and the model gets a fraction of it. A catalogue has millions of items. A good ranking model costs roughly a millisecond per item.

1,000,000 items x 1 ms  =  1,000 seconds per request
Budget for ranking      =  tens of milliseconds

Four orders of magnitude. No hardware closes it, so the architecture changes — and the shape it changes into is a cascade where each stage examines fewer items with a better model.

The stages, and what each is for

Each stage has a different objective, and confusing them is the most common error in this design.

The fourth row is the one people leave out of the diagram. Diversity, freshness and business rules are properties of the set, not of any individual item — and a ranker scoring items independently cannot express them. That is why re-ranking is a separate stage rather than a few more features.

Why pre-ranking exists

The two-stage version goes straight from a hundred thousand candidates to the heavy ranker. The problem is arithmetic again: if the ranker costs a millisecond an item and you have thirty milliseconds, it can score about thirty items. Retrieval returning ten thousand is useless if only thirty survive to be scored properly.

So a stage sits between them: a lightweight model that approximates the ranker cheaply enough to score thousands. It is deliberately worse than the ranker — simpler architecture, fewer features — and its job is not to be right, but to be right about what the ranker would have liked.

The compounding ceiling

The property that governs the whole design, and the reason to measure per stage.

Each stage can only work with what the previous one gave it. So the ceilings multiply:

Candidate generation recall   0.90
x Pre-ranker recall           0.95
x Ranker quality              (whatever it is)
                              -----
Best possible end-to-end      0.855 of the ideal

Two consequences worth stating.

The earliest stage matters most. A ten-point recall loss at candidate generation cannot be recovered by any amount of ranking improvement. This is why "measure retrieval recall before tuning the ranker" keeps being the right first move.

Adding a stage adds a ceiling. Pre-ranking buys the ability to consider more candidates and costs a little recall. That is usually a good trade — considering ten thousand candidates at 95% pre-ranker recall beats considering thirty at 100% — but it is a trade, and pretending the extra stage is free is wrong.

When you do not need four stages

Being honest about it, because proposing four stages for a small system is over-engineering.

Under about ten thousand items, score everything with the good model. No cascade, no recall ceiling, no consistency problem. Simplest and best.

Under about a million, two stages usually suffice — retrieval and ranking — because the ranker can handle the few hundred that retrieval returns.

Pre-ranking earns its place when retrieval legitimately needs to return thousands rather than hundreds, which happens when the catalogue is large and recall at small k is poor.

Re-ranking earns its place the moment set-level properties matter, which is almost immediately in any consumer product — nobody wants ten results from the same seller.

Next: where the candidates come from, and why one source is never enough.