Downsampling and the Correction

In one line: keeping every negative is wasteful, dropping most of them is standard, and forgetting to undo it makes every prediction wrong by a factor you could have computed exactly.

Why downsample at all

With a click rate well under one percent, a day of impressions is overwhelmingly negatives that all look alike. Training on all of them costs enormous compute for very little additional signal — the thousandth non-click on a given ad teaches less than the first.

So: keep every positive, keep negatives with probability w. A rate that brings the classes closer to balanced is a common starting point, though the right value is empirical.

The gains are real. Training is faster, so you can retrain more often — and in a system where freshness beats sophistication, that alone justifies it.

What it breaks

Downsampling changes the base rate the model observes. If the true click rate is 0.1% and you keep 1% of negatives, the training data's apparent click rate is roughly 10%.

The model learns that world faithfully and predicts on that scale. Every output is systematically far too high — not noisily wrong, but wrong by a specific factor.

Note what this does not break: ordering is unaffected. AUC is identical before and after. So a team monitoring only AUC sees nothing wrong while every price in the system is computed from an inflated probability.

That is the cleanest example in this chapter of why AUC is insufficient.

The correction

The fix is analytic, not empirical. Given p, the model's output on downsampled data, and w, the negative sampling rate, the calibrated probability is:

q  =  p  /  ( p + (1 - p) / w )

Sanity-check the shape rather than memorising it. With w = 1 — no downsampling — the denominator is p + (1 - p) which is 1, so q = p and nothing changes. As w gets small, (1 - p)/w grows large, the denominator grows, and q shrinks — which is right, because aggressive downsampling inflates predictions and the correction must pull them back down.

This is the standard treatment from the published Facebook ads work, and it is the sort of detail that reads as production experience rather than coursework.

The traps

Four, and each has produced a real incident somewhere.

The rate changes and the correction does not. Someone tunes w for training speed and the serving-side constant is left alone. Every prediction is now wrong by the ratio of the two rates. The rate belongs with the model artifact, not in a config file that a different team edits.

Per-segment sampling rates. Downsampling more aggressively on high-volume segments is reasonable and means w varies by segment. The correction must then be applied per segment with the matching rate, and a single global constant silently mis-corrects everything.

Correcting twice. The correction is applied in the training pipeline's evaluation and again at serving. Predictions are now too low, which looks like a model quality problem and gets debugged as one.

Downsampling positives. Almost always wrong. Positives are the scarce, informative examples; there are already too few. Downsampling is a technique for the abundant class only.

The alternative: importance weighting

Rather than dropping negatives, keep them and weight them. Equivalent in expectation, no correction needed, no sampling variance — and it does not save any compute, which was the point.

The middle ground used in practice: downsample to make training tractable, then weight the retained negatives by 1/w so the loss reflects the original distribution. This gives most of the speed benefit and keeps the loss unbiased, at the cost of higher gradient variance since a few examples now carry large weights.

Why accuracy is meaningless here

Worth one paragraph, because it comes up.

At a 0.1% click rate, a model predicting "no click" for every impression is 99.9% accurate and completely useless. Accuracy is not a slightly poor metric on imbalanced data — it is actively misleading, because the trivial model beats almost anything.

The metrics that survive are log loss, which is a proper scoring rule and rewards well-calibrated probabilities; AUC or PR-AUC for ordering, with PR-AUC being the more informative under this much imbalance; and calibration ratios per segment.

Where the sampling meets the auction

The point that ties this lesson to the rest of the chapter.

The corrected probability is what enters eCPM = bid x pCTR x quality. An uncorrected model does not produce randomly wrong prices — it produces systematically inflated eCPMs across the board, which changes which ads win relative to any advertiser whose pricing does not go through the same path, and makes every reserve price comparison meaningless.

Reserve prices are absolute amounts. Comparing an inflated eCPM against a fixed reserve means the reserve is effectively far lower than intended, and slots that should have gone unfilled are sold to ads nobody wants to click.

Next: the labels that have not arrived yet.