Multi-Objective Ranking

In one line: the ranker predicts several things and something else decides what they are worth, and keeping those two jobs separate is what makes the system steerable.

Several heads, one model

Since no single objective survives optimisation, the ranker predicts a set of them. A shared trunk learns the user-item representation; separate heads read off different predictions.

Two reasons this beats separate models. The trunk is shared, so one forward pass produces every prediction — decisive when the budget is fifteen milliseconds for a hundred items. And the heads regularise each other: a sparse objective like survey score learns better beside a dense one like click, because the shared representation is supported by the dense signal.

Note the last head. Negative-signal prediction is easy to omit and expensive to omit — "not interested", hide and report are the strongest quality signals a user ever sends, and they are the only ones that are unambiguous.

The value model

The heads produce predictions. Something must turn them into one ordering.

score = w1*P(click) + w2*E[dwell] + w3*P(complete)
      + w4*P(save) + w5*predicted_satisfaction
      - w6*P(hide or report)

Simple, and doing more than it looks. It is where the product's values are written down, in numbers, in one place. What is a save worth relative to a click? How much predicted dissatisfaction cancels a likely click?

Keeping this separate from the model has a practical consequence that is worth stating in an interview: you can change what the system optimises without retraining anything. The heads predict facts about user behaviour, which do not change when your priorities do. Fold the weights into the loss and every priority change becomes a training run.

Setting the weights

Nobody derives them. There are three ways they get set, in increasing sophistication.

By hand, then A/B. Pick values that seem reasonable, ship, measure guardrails, adjust. Crude, and it works, and it is where every system starts.

By calibrating to a target. Decide what a save is worth in units of watch time — say a save is worth five minutes because saved items get returned to — and set the ratio from that. Better, because it forces an explicit claim you can then check.

By optimising against a long-term outcome. Treat the weights as parameters and search them against retention or satisfaction measured over weeks. Expensive, slow, and the only method that targets what you actually want. This is where published work on learned ranking functions sits — tuning the combination against long-term satisfaction rather than the short-term signals themselves.

Whichever you use, the weights need an owner and a review cadence. Weights set once during a launch and never revisited are a common and invisible source of drift, because the product changes around them.

When the objectives disagree

They will, and the disagreement is information.

An item with high click probability and high hide probability is clickbait, and the negative head is the only thing that catches it. An item with low click probability and very high completion probability is niche and valuable to the few who start it. High dwell with low satisfaction is content that holds attention without earning it.

A single-objective system cannot represent any of these distinctions, which is the real argument for multiple heads. It is not accuracy — it is expressiveness.

Pareto, and why the framing helps

There is no ordering that maximises every objective at once. Improving predicted satisfaction usually costs some click-through; improving diversity usually costs some short-term engagement.

The set of orderings where you cannot improve one objective without hurting another is the Pareto frontier. The weights choose a point on it.

This framing is useful in an interview for one reason: it converts an argument about whether a change is "good" into a question about which point on the frontier the product wants. A change that trades 2% click-through for 5% satisfaction is not better or worse — it is a different choice, and someone with product authority should be making it.

Calibration, and when it matters

If the score only orders a slate, calibration is irrelevant — a monotone transform changes nothing.

It matters as soon as the number is compared to anything else. Blending with an ad auction, applying a confidence threshold to decide whether to send a notification, or combining heads whose scales differ all require the predictions to mean something absolutely. A probability that says 0.3 should be right about 30% of the time.

So: calibrate when the score leaves the slate, and do not bother when it does not. Being able to say which case you are in is worth more than knowing the techniques.

Next: the reason none of these labels mean what they appear to.