Calibration Is Load-Bearing

In one line: a calibrated model's predicted 2% happens 2% of the time, and in an ads system that property is the difference between correct pricing and confident fiction.

The measurement that matters

Aggregate calibration is easy and nearly useless. Sum the predictions across all impressions, compare to the actual click count, and the ratio should be one.

calibration ratio  =  sum(predicted)  /  sum(observed)

A ratio of 1.0 across all traffic is consistent with a model that is 2x high on mobile and 2x low on desktop. The errors cancel in the sum, the dashboard is green, and both segments are being priced wrongly in opposite directions.

So calibration is only meaningful per segment. The segments that matter are the ones the auction distinguishes: advertiser, vertical, placement, device, position, and — critically — the pCTR band itself.

That last segment is the subtle one. Bucket predictions by their value — everything predicted between 1% and 2%, between 2% and 3%, and so on — and check each bucket's observed rate. A model can be well calibrated in the middle of its range and badly off in the tails, and the tails are where the expensive decisions are: the highest-pCTR predictions are the ones winning the most valuable slots.

What breaks it

Four causes, and they need different fixes.

Downsampling, uncorrected. Training on downsampled negatives shifts every prediction upward by a known factor. This one is analytic and gets its own lesson.

Distribution shift. The model was trained on last week and traffic has moved — a new campaign launched, a seasonal shift, a placement changed. Ordering can survive a shift that ruins the scale.

Selection bias in the training data. You only observe clicks on ads that won auctions. Ads that never win are absent from training, so the model's estimate for them never improves — the auction is a filter on its own training data, which is the closed loop with money attached.

Optimising the wrong loss. Training a ranking objective, or using a loss that only cares about ordering, produces a model with no reason to be calibrated at all. Log loss is a proper scoring rule and rewards calibration directly; a pairwise ranking loss does not.

Fixing it after the fact

When the model is well-ordered and badly scaled, a monotone post-hoc mapping fixes the scale without touching the ordering — Platt scaling for a smooth parametric correction, isotonic regression when the distortion is not monotone-smooth and you have enough data.

Two things specific to ads.

Calibrate per segment, not globally. A single global mapping cannot fix a model that is high on mobile and low on desktop — it will make both wrong by the average. Fit the mapping per segment, and be careful about segments with little data, where the mapping is fitting noise.

Recalibrate continuously. Calibration decays faster than ranking quality, because the traffic mix moves. A model whose AUC has not degraded may have drifted well out of calibration in a day. Recalibrating on recent data is cheap — it is a one-dimensional fit — and it is one of the highest-value maintenance jobs in an ads stack.

The metric set

Three numbers, and reporting only the first is the common failure.

AUC and log loss can move in opposite directions, and when they do, log loss is the one to believe for an ads system — a model that orders slightly worse and prices correctly is better than the reverse.

A useful extra: relative information gain against the baseline of always predicting the average click rate. It puts log loss on an interpretable scale, since raw log loss values on a 0.1% base rate are all small and hard to compare.

The failure that hides

The specific scenario worth being able to describe, because it is what an interviewer is checking for.

A new advertiser's ads are systematically under-predicted — the model has little data on them and regresses toward a low prior. So their eCPM is understated, they lose auctions, they get few impressions, and almost no click data comes back. The under-prediction persists indefinitely.

From the platform's side everything looks fine: aggregate calibration is good, AUC is stable, revenue is on plan. From the advertiser's side the platform does not work, and they leave.

The fix is not a modelling change. It is the exploration budget — guaranteeing new advertisers some impressions regardless of their estimated eCPM, so the estimate has data to correct itself. Which is cold start again, with a bill attached, and it gets its own lesson.

Next: the features, and why there are billions of them.