The Pre-Ranker and Ranking Consistency

In one line: the pre-ranker's job is not to be accurate, it is to be accurate about what the ranker would have chosen — and those are different objectives that diverge in a way nothing else in the pipeline catches.

What it is

A deliberately lightweight model sitting between candidate generation and ranking, taking roughly ten thousand candidates down to a hundred. Simpler architecture, fewer features, and enough inference efficiency to score thousands of items inside a few milliseconds.

Its constraints pull against each other, which is what makes it interesting:

It must be fast enough for ten thousand items. That rules out the ranker's cross features, its embeddings lookups, and most of its depth.

It must approximate the ranker. A cheap model that ranks differently is worse than useless — it discards candidates the expensive model was going to pick.

Consistency, not accuracy

Here is the failure that makes this stage worth a lesson.

Suppose the pre-ranker is genuinely good at predicting clicks — better, say, than a naive baseline. And suppose it weights recency heavily while the ranker weights personalisation heavily. Both are reasonable models. Both would score well on their own offline evaluation.

The pre-ranker forwards recent items. The ranker wanted personalised ones. Every request, the ranker is handed a set it would not have chosen, and its output is the best of a bad hundred.

Nothing in the system reports this. The pre-ranker's accuracy looks fine. The ranker's accuracy on the items it received looks fine. End-to-end quality is poor and neither component is obviously at fault.

How they drift apart

They start consistent and separate over time, usually for one of four reasons.

The ranker is retrained and the pre-ranker is not. The most common cause and the most avoidable. The ranker learns something new about what users want; the pre-ranker keeps forwarding what the old ranker wanted.

Different features. The pre-ranker cannot afford the ranker's features, so it is approximating a function it cannot fully see. Some divergence is structural and irreducible; the question is how much.

Different training objectives. If the pre-ranker is trained on clicks and the ranker on conversions, they are optimising different things and will disagree about anything where those diverge — which is a lot.

Different training data. The pre-ranker sees candidates; the ranker sees the pre-ranker's output. Train each on what it observes and the pre-ranker learns from a broader, differently-distributed set.

The first is the one to fix by process: treat the pre-ranker and ranker as a unit that is retrained and deployed together. Deploying one without the other is a silent quality regression, and it looks like nothing on any dashboard.

Distilling the ranker

The cleanest way to make them consistent by construction: train the pre-ranker to predict the ranker's scores rather than to predict user behaviour.

The target becomes the ranker's output on the same candidate, so the objective is explicitly "agree with the expensive model". Consistency stops being something you hope for and becomes what you optimised.

The fourth row is an underrated practical advantage. Behavioural labels exist only for items that were shown, so the pre-ranker is trained on a biased slice of the very candidate set it is supposed to filter. Ranker scores can be generated for any candidate offline, including the ones nobody ever saw — which is exactly the distribution the pre-ranker operates over.

The third row is the honest cost: a distilled pre-ranker can never exceed the model it is imitating. For this stage that is the correct trade, because it is not trying to be the best model — it is trying to be a cheap, faithful proxy for one.

When to skip it

Do not add a pre-ranker by default. It costs a stage, a model, a training pipeline, a consistency metric, and a coupled deployment.

It earns its place when candidate generation genuinely needs to return thousands — a large catalogue where recall at a few hundred is poor. If retrieval can return three hundred good candidates and the ranker can score three hundred in budget, there is nothing for a pre-ranker to do except add a ceiling.

Next: what the ranking model can see that nothing upstream can.