The Objective Is the Design Decision

In one line: a recommender is a machine for maximising a number, so the number you pick is the design, and everything after it is implementation.

The question that comes before the architecture

Asked to design a recommendation system, most candidates start drawing boxes. Candidate generation, ranking, a feature store. That answer is available to anyone who has read a blog post, and it skips the only decision that is genuinely yours.

What is this system for? A video platform maximising clicks builds something different from one maximising watch time, which builds something different again from one maximising a satisfaction survey. Same boxes, different system.

The industry learned this the expensive way, publicly.

Clicks, and what they bought

Early recommenders optimised click-through rate. It is the obvious choice: abundant, immediate, unambiguous.

It also produced exactly what it asked for. A model rewarded for clicks learns that a misleading thumbnail earns a click, that an outrageous title earns a click, and that a user who clicks and leaves after two seconds has been served perfectly. The metric was satisfied and the product was worse.

This is Goodhart's law with a recommendation engine attached. The moment a measure becomes a target, it stops measuring what it used to.

Watch time, and what that bought

YouTube's 2016 recommender paper made the shift explicit: the system optimises expected watch time rather than click probability, on the argument that click-through favours deceptive content while watch time is closer to what the viewer actually wanted.

It was a genuine improvement and it had its own failure. Watch time rewards length. It rewards the autoplay chain. It cannot distinguish a viewer who is absorbed from a viewer who is stuck, and it treats an hour someone regrets as identical to an hour they valued.

Every proxy has this property. It correlates with the thing you want, right up until a model optimises hard against it, at which point it finds the region where the correlation breaks.

Satisfaction, and why it is hard

The next move is to ask people. Surveys give a signal that is much closer to the real objective — did you find this valuable? — and it is expensive and sparse. You cannot survey every impression.

So the survey becomes a label for a model: predict the survey score from behaviour, then use the prediction as one signal among several. Recent published work on ranking functions describes exactly this shape — short-term behavioural predictions combined into an estimate of long-term satisfaction.

Note what happened structurally. The objective got further from the data, so you now need a model to bridge them, and that model has its own error. That is the price of a better objective, and it is worth paying.

The pattern, stated once

That trade — closer to the truth, further from the data — is the sentence to have ready. It generalises past recommendation to every system where the thing you can measure is not the thing you want.

No single number survives

Which leads to where production systems actually land: not one objective, but several, predicted separately and combined.

A ranker predicts click probability, expected dwell, completion likelihood, save or share probability, and a predicted satisfaction score. Those are separate heads on one model. A weighted combination produces the score that orders the slate.

The weights are the product decision. They encode how much a save is worth relative to a click, how much a completed view is worth relative to a long one. They belong to whoever owns the product, not to whoever tunes the model — and a candidate who calls them a hyperparameter has misunderstood what they are.

Guardrails: the metrics you are not optimising

One more piece, and it is what separates a designed system from an optimised one.

You choose a small number of objectives to maximise. You choose a larger number of metrics that must not get worse. Retention, session count, creator diversity, reported dissatisfaction, time to first meaningful action.

Guardrails exist because optimisation finds the cheapest path to the target, and the cheapest path frequently runs through something you care about but did not name. Naming them in advance converts a discovery into an alarm.

Next: scoping the product, and the stakeholders whose interests do not align.