Where Labels Come From
In one line: the model you can build is decided by the labels you can get, so asking where the label comes from is a higher-value question than asking which architecture to use.
The five sources
Every supervised system needs an answer, and the options differ enormously in volume, cost and honesty.
| Label source | Volume | The catch |
|---|---|---|
| Explicit user signal | Tiny | Ratings and thumbs are rare, and the users who give them are unrepresentative |
| Implicit behaviour | Enormous | A click is not approval; it is attention, and it is biased by position |
| Human annotation | Bounded by budget | Needs a rubric, and inter-annotator agreement becomes a metric you must track |
| Weak or programmatic | Large | Rules generate labels fast and encode their author's blind spots at scale |
| Downstream outcome | Moderate | The most honest label, and it arrives days or months late |
Most production systems use several at once, and saying which one you are using for what is the mark of a considered answer. A common and defensible split: implicit behaviour to train, human annotation to evaluate, and downstream outcome to decide whether the whole thing was worth it.
Implicit behaviour is abundant and lies in a known direction
A click means the user found the item worth investigating from a thumbnail and a title. It does not mean the item was good. Those two get conflated constantly, and the gap is exactly the space in which clickbait wins.
The position bias is the bigger problem. The top result is clicked far more than the fifth partly because it is on top, not because it is better. Train naively on click data and the model learns to reproduce your current ranking, since that is what the click distribution mostly encodes.
The standard countermeasure is to weight each observation by the inverse probability that its position was examined, so a click at position ten counts for more than a click at position one. It is worth naming even if you do not derive it — it signals you know that raw click data is not ground truth.
Human annotation buys quality and introduces a new system
Choosing human labels is choosing to build a labelling pipeline: a rubric, a pool of annotators, a training process, a quality-control sample, and a disagreement-resolution rule.
Two numbers matter and get forgotten. Inter-annotator agreement tells you the ceiling on model performance — if two trained humans agree only 70% of the time on whether a comment is abusive, a model scoring 85% against one annotator is not measuring what you think. And cost per label decides whether the approach scales past the prototype.
The honest label arrives late
This is the structural constraint that shapes most real designs.
For fraud, the true label is a chargeback that can arrive up to 90 days after the transaction. For a recommendation, whether the user was glad they watched is knowable only much later, if ever. For a support bot, whether the issue was actually resolved shows up as the absence of a follow-up contact a week later.
The design consequence is a split that is worth stating explicitly: train on the fast proxy, evaluate against the slow truth.
That has a second-order effect people miss. Because the true label lands months later, your evaluation of today's model is always a judgement about a model you have already replaced. Any system with delayed labels needs a way to attribute an outcome back to the model version that caused it — which means logging the model version and the features used with every decision, not just the decision.
The label-delay trap in fraud
The classic version: you deploy a fraud model, block a class of transactions, and those transactions never generate chargebacks. Your fraud rate improves. But you now have no labels for exactly the population you are most interested in, because you prevented the outcome that would have labelled it.
The countermeasure is to let a small, budgeted fraction of borderline cases through deliberately, purely to observe what happens. That is not a modelling decision — it is an architecture and policy decision, and it belongs in the design.
Cold start, answered properly
Every AI design gets asked what happens for a new user or a new item. "We use a default" is the weak answer. The strong one separates the two cases, because they are not symmetric.
The asymmetry is the answer. A new item has content, so the fix is a model that can score from content. A new user has nothing, so no model can help, and the fix is a policy: show what is broadly good, use whatever context arrives for free, and spend a few impressions deliberately learning about them.
That last part — spending impressions to learn — is worth naming as a cost. Exploration is not free; it trades short-term metric for long-term knowledge, and the size of that budget is a decision the interviewer may well probe.
Key takeaway
Ask where the label comes from before you ask which model to use, because the answer constrains everything downstream. Implicit behaviour is abundant and biased by position; human annotation buys quality and obliges you to build a labelling system whose inter-annotator agreement caps your achievable accuracy. The honest label always arrives late, so train on the fast proxy and evaluate against the slow truth — and log the model version with every decision, because by the time the truth lands you have already shipped something else.
Next: the features that exist in your warehouse and not at decision time.