From a Business Goal to a Number
In one line: you are handed a goal in English and you have to produce a number a machine can maximise, and much of the interview is whether you notice what falls out at each step.
The translation chain
Nobody asks you to minimise cross-entropy. They ask you to help people find things they want to watch. Between those two sentences sit three translations, and each one loses something specific.
Naming the loss at each arrow is most of the skill. A candidate who walks the chain and says what is given up sounds like someone who has shipped a system. One who jumps straight to the loss sounds like someone who has only trained models.
Arrow one: the goal is about people you cannot see
"People find things they want" includes people who tried the product once, failed, and left. They generate no data. The objective you write is therefore about the users you retained, which is a survivorship-biased sample of the users you wanted.
This is the quiet reason systems get better at serving the people they already served well. It argues for segment-level reporting from the start, which we return to at the end of this lesson.
Arrow two: satisfaction is not observable
You want to know whether the user was glad. You can observe whether they clicked. Those are different quantities and the gap between them is where most design failures live.
Arrow three: the loss has a shape
A loss function is not a free-form wish. It has to be differentiable, computable at scale, and defined for every training example. Objectives that resist that shape — "be helpful", "be diverse", "be fair" — get approximated or moved out of the loss entirely into a re-ranking layer or a guardrail. Knowing which of your goals will not survive the trip into a loss function is a genuinely senior observation.
The proxy is the dangerous step
The middle arrow carries the most risk, because a proxy gets chosen for being measurable, not for being right.
Consider the ladder a video recommender typically climbs, in the order teams usually climb it:
| Proxy | What it gets right | What it rewards that you did not intend |
|---|---|---|
| Click | Abundant, immediate, unambiguous | Thumbnails that oversell — the click happens before the value does |
| Watch time | Closer to value than a click | Long content over good content; autoplay loops |
| Completion rate | Accounts for length | Short content; penalises anything worth pausing |
| Explicit rating | Actually asks the user | Almost nobody answers, and the ones who do are unrepresentative |
| Return visit next day | Hard to game in a single session | Slow, sparse, and confounded by everything else in the product |
Two patterns are worth extracting. Every proxy is gameable in a direction, and the system will find that direction because finding directions is what optimisation is. And the proxies get better as they get slower — the most honest signals are the ones that arrive too late to train on.
Goodhart's law is a design constraint, not a quotation
The usual phrasing is that when a measure becomes a target, it stops being a good measure. Stated that way it sounds like philosophy. In an AI system it is a mechanical prediction about what your model will do next quarter.
Platforms optimise engagement because engagement correlates with revenue. Optimising it hard produces outrage-driven and compulsive content, which maximises the measured quantity while making users measurably worse off. The proxy and the goal come apart precisely because the system got good at the proxy.
The interview-relevant form is sharper: a stronger model makes a bad proxy worse, not better. If your metric is subtly wrong, improving the model moves you faster in the wrong direction. So the order of operations is to check what the loop optimises before you make the loop stronger.
Guardrail metrics
The fix is not to search harder for a perfect proxy. There isn't one. The fix is to pair the metric you optimise with metrics you refuse to damage.
The distinction that matters: a guardrail is not a secondary objective you also optimise.
Adding a metric to the objective changes what the model learns — and it forces you to pick weights, which is a decision you usually cannot justify. Adding a guardrail changes only what you are willing to ship. It costs no modelling complexity and it catches exactly the case you are worried about.
| Add to the objective | Add as a guardrail | |
|---|---|---|
| Changes | What the model learns | What you are willing to launch |
| Requires | A weight you must justify | A threshold you must agree |
| Cost | Modelling complexity, retraining | A line in the experiment analysis |
| Use when | The concern is continuous and central | The concern is a floor you must not cross |
Naming two or three guardrails unprompted is among the cheapest strong signals available in this round. Most candidates name a primary metric and stop.
Choosing thresholds you can defend
A guardrail without a number is a wish. Two ways to pick one that survives a follow-up question.
Anchor to a known-acceptable state. Latency p99 must not exceed today's value, because today's value is evidently tolerable. This is defensible without any new information.
Anchor to a decision. Complaint rate must not rise more than 5% relative, because above that support costs exceed the revenue the change earns. This is stronger, because the number comes from a trade-off rather than from comfort.
Counting the people you cannot see
Return to arrow one, because it has a design consequence rather than only a caveat.
Your training data describes users the current system already serves. People it serves badly leave, and leaving produces no label. So they disappear from the data that trains the replacement, and the replacement optimises for the survivors.
The result is a system that is measurably better on average and worse for a segment, with nothing in the aggregate to show it.
The countermeasure is not clever: report the primary metric and the guardrails by segment, and decide in advance which segments you will look at. New users, low-activity users, each major locale, and each major device class are a reasonable default set.
Doing this in the design rather than in a post-mortem is what the round is testing.
Key takeaway
Turn the goal into a number in explicit steps and say what each step costs. The proxy is where designs fail, because it is chosen for being measurable and then optimised until it comes apart from the goal — and a stronger model makes a bad proxy worse. Pair the metric you optimise with guardrails that block a launch rather than folding them into the objective, give each guardrail a defensible threshold, and report by segment so a win on the average cannot hide a loss on a group.
Next: where the labels that make any of this possible actually come from.