The Funnel, Sized

In one line: the cascade is standard, so the signal is not that you know it — it is whether your numbers hold together.

The shape, and where it came from

Ten million items, tens of milliseconds, a heavy model. You cannot score everything, so you spend a cheap model on many items and an expensive one on few. The 2016 YouTube recommender paper is the canonical published version: candidate generation narrows millions to hundreds, then ranking orders them with a much richer model.

Drawing it earns nothing. Sizing it earns everything.

Where the numbers come from

Each is derived, not remembered, and being able to derive them live is the point.

How many candidates? Bounded by what the pre-ranker can score in its slice of the budget. If a pre-ranker scores a candidate in about 5 microseconds and you have 5 milliseconds, that is roughly a thousand. The number is a consequence of the budget, not an input to it.

How many reach the ranker? Bounded the same way. A ranker with cross features costs perhaps 100 microseconds per item; 15 milliseconds buys about 150. Round to a hundred.

How deep is the slate? Product, not engineering. A feed showing ten before the fold needs enough beyond it for scrolling, so you score more than you show.

What is the budget? Work backwards from the page. If the surface must render in 200ms and the recommender is one of five calls, you have perhaps 50ms end to end — and feature fetching, not model inference, is usually the largest slice.

The two questions the diagram cannot answer

"What does each stage cost you?"

Every stage is a filter, and every filter has a recall cost. Candidate generation is the expensive one: an item not retrieved cannot be ranked, so its recall failure is invisible and permanent. Nothing downstream can recover it, and no online metric shows it, because you never learn what would have happened.

That asymmetry is why candidate generation is measured on recall and ranking on ordering quality. Different stages, different metrics, for a structural reason.

"Why does the pre-ranker exist?"

Only to make the budget work. It is a strictly worse model that sees strictly fewer features, and its job is to discard 90% of candidates without discarding what the ranker would have chosen.

Which sets up the failure mode worth naming: if the pre-ranker disagrees with the ranker, it throws away the ranker's picks before the ranker sees them. The pre-ranker is not trying to be accurate about user preference. It is trying to agree with the ranker, which is why it is often trained on the ranker's own scores rather than on labels — the ranker's ordering is the target.

Where the time actually goes

Candidates consistently over-budget model inference and under-budget everything else.

Feature fetch dominates because it is a hundred candidates times many features, and the naive implementation issues one lookup per candidate. Batched multi-get is the fix, and it is a store-selection criterion rather than an optimisation.

When not to build a funnel

Worth saying, because it is the cheapest way to show judgement.

Under roughly ten thousand items you can score everything with the full model and skip candidate generation entirely. No ANN index, no pre-ranker, no recall problem, no two-stage disagreement — and better results, because the heavy model sees every item.

The funnel is a concession to scale. Proposing it when the catalogue does not require it is the same error as proposing a distributed database for a hundred megabytes.

What is different here from generic retrieval

Two things, and both get their own lessons.

The candidate stage is not one retriever. It is several sources blended, because no single source covers the space — and that is a recommendation-specific structure that a search cascade does not have.

And the ranker does not predict one number. It predicts several, which are combined by weights that encode a product decision.

Next: why the candidate stage is a portfolio rather than a retriever.