Delayed Feedback

In one line: a click arrives in seconds and a conversion arrives in hours or days, so at training time you cannot tell a true negative from a positive that has not happened yet.

Two different label problems

Clicks are fast. A user either clicks within the impression's lifetime or does not, and a few minutes is enough to label it. Delay is a minor complication.

Conversions are slow and unboundedly so. Someone clicks an ad for a sofa today, thinks about it, and buys on Saturday. A purchase five days later is genuinely attributable to that click, and at the moment you label the training data it has not happened.

So a conversion label is never final. It is "no conversion yet", and the difference between that and a true negative is the whole problem.

The trade nobody escapes

Two knobs pulling against each other.

Wait longer and the labels are more accurate — fewer positives mislabelled as negatives. And the model is trained on older data, in a domain where new campaigns and creatives appear constantly.

Wait less and the model is fresh and systematically under-predicts conversion, because a growing share of the positives had not landed when the label was taken.

Neither end is acceptable. A week-old model in an ads system is badly stale; a one-hour label window mislabels most conversions as negatives.

The attribution window

The product decision underneath the modelling one, and it is a decision rather than a fact.

A conversion is credited to a click within some window — commonly a few days for clicks, shorter for view-through. That window is a business rule negotiated with advertisers, not something you derive.

Two consequences for the design worth naming:

Labels are only complete once the window closes. So there is always a trailing period where the data is partial by construction, and any metric computed over it is biased low until the window passes.

Changing the window changes the labels retroactively. Extending from seven days to thirty adds positives to historical data and shifts every metric. That is a comparison-breaking event, and it needs to be treated like a schema migration rather than a config tweak.

Making partial labels usable

The useful reframing: this is not a classification problem with noisy labels. It is a time-to-event problem, and an unconverted click is a censored observation — you know the conversion had not happened by the time you looked, not that it never will.

That framing makes the data usable rather than merely flawed. Two components, learned jointly:

Will it convert at all? The probability the click eventually converts.

How long will it take? A distribution over the delay.

With both, an observation labelled negative after two hours can be used correctly: the model knows that most conversions for this kind of click take longer than two hours, so two hours of silence is weak evidence rather than strong.

This is the shape of the published delayed-feedback modelling work, and it is the answer to give when asked how to train on incomplete labels. The simpler alternatives — waiting for the window, or accepting the bias — are both worse, and being able to say why is the point.

Simpler things that work

Not every system needs survival modelling, and the pragmatic options are worth having.

Two models. One on fresh, partially-labelled data for responsiveness; one on fully-matured data for accuracy. Blend them, weighting toward the mature model as the window closes.

Importance weighting. Weight recent examples by the probability that their label is still going to change, so uncertain negatives count for less.

Correct at the aggregate. Learn the historical delay distribution and scale up recent conversion counts by the expected fraction still to arrive. Crude, and it fixes the calibration of aggregates even if it does nothing per-example — which for pacing and reporting is often what you actually need.

Predict clicks, not conversions, where you can. If the advertiser pays per click, none of this applies. The delayed-feedback problem is imported by the payment model, and it is worth checking whether you have actually taken it on.

The compounding problem

The reason this matters more than it first appears.

Delayed labels feed a model that under-predicts conversion. Under-predicted conversion lowers eCPM for conversion-optimised advertisers. They win fewer auctions, so they get fewer clicks, so fewer conversions come back, so the estimate stays low.

The feedback loop again — and here it is selective, hitting the advertisers whose conversions are slowest. A luxury purchase with a two-week consideration cycle is systematically disadvantaged against an impulse buy, not because it is worth less but because its evidence arrives later.

That is a real fairness issue in the marketplace, and naming it is a strong signal. The mitigation is the delay-aware model: without it, the system quietly favours fast-converting verticals.

Next: why these models are trained continuously rather than in batches.