Evaluation and Serving

In one line: the model's offline metric and the platform's revenue are connected by an auction, and that connection is not monotone.

The offline set

Three numbers, and the ordering matters.

Log loss first, because it is a proper scoring rule — its minimum is at the true probability, so it rewards calibration directly rather than incidentally. Report it as relative information gain against the always-predict-the-base-rate baseline, since raw log loss on a fraction-of-a-percent click rate is a small number that is hard to compare across datasets.

Calibration ratio, per segment. By advertiser, vertical, placement, device, position, ad age and pCTR band. The aggregate is nearly useless on its own.

AUC or PR-AUC for ordering. Useful and insufficient, for the reason the whole chapter turns on: it is invariant to the scale that the auction depends on. PR-AUC is the more informative of the two under this much imbalance.

Why offline improvement does not imply revenue

The gap that makes this system harder to evaluate than a ranker, and it has four separate causes.

The auction is not monotone in prediction quality. Better pCTR estimates change who wins, which changes prices, which changes revenue in a direction that depends on the competitive structure of each auction. A more accurate model can reduce short-term revenue by correctly lowering the eCPM of ads that were previously over-valued — which is the right outcome and looks like a regression.

Advertisers respond. Change the system and bids, budgets and creatives change over the following weeks. Offline evaluation holds the advertiser's behaviour fixed; production does not.

Pacing interacts. A better model changes win rates, which changes spend rates, which makes the pacer throttle differently, which changes which impressions are bought. The second-order effect can be larger than the first.

The evaluation data came from the old policy. The same closed loop as recommendation: you only observe clicks on ads that won auctions under the previous model.

The experiment

An ads A/B test has a complication that most do not, and it is worth raising unprompted.

Budgets are shared across arms. An advertiser has one daily budget, and if treatment causes them to win more auctions, they exhaust that budget faster — which affects their behaviour in the control arm too. The arms are not independent, and the interference is through a shared resource rather than through a shared catalogue.

The standard mitigations are budget-splitting — giving each arm a proportional share of every advertiser's budget, so the arms are genuinely separate marketplaces — or randomising by advertiser rather than by user, which removes the interference and costs a great deal of statistical power.

Two more:

Auction-level interference. Treatment ads competing in the same auction as control ads means each arm changes the other's clearing prices. Splitting the marketplace properly means splitting the auction, which is a substantial engineering commitment.

The horizon is long. Advertiser responses take weeks. A two-week test measures the mechanical effect and misses the behavioural one entirely, which is why platform-level changes to auction or pricing usually need long holdbacks rather than short tests.

What to measure

Revenue alone is the trap, because it is trivially increased by degrading the user experience — more ads, more prominently — and the cost arrives later.

"At a fixed ad load" in the first row is doing the work. Comparing revenue between arms that showed different numbers of ads compares ad load, not model quality.

The marketplace tier is the one that separates a senior answer from a staff one. Rising advertiser concentration — a shrinking set of advertisers winning a growing share — is the ads equivalent of catalogue narrowing, and it is invisible in revenue right up until the concentrated advertisers have market power over you.

Serving

The budget is tight and the shape is a funnel, for the same reason as everywhere else in this course.

Three things dominate the budget, and only one of them is the model.

Embedding lookups. Sharded across machines, so a single inference is a distributed gather. Batching the lookup across candidate ads is the highest-value optimisation available — the same batched-multi-get point that recurs throughout this course.

Feature fetch. Advertiser state, budget remaining, frequency caps, user history. Fetch what does not depend on the candidate in parallel with candidate selection.

The auction itself. Cheap arithmetically, and it depends on every candidate's score, so it cannot start until scoring finishes. It is a synchronisation point in an otherwise parallel pipeline.

Degradation

Ads have a failure mode most systems do not: showing nothing is an acceptable outcome. An empty slot loses revenue for one impression and harms nobody.

That makes the ladder simpler and more honest than a feed's:

Full pipeline. Then a cached pCTR per ad rather than a live model. Then an advertiser-level historical rate. Then bid-only ranking, which is what an auction was before any of this existed. Then no ad.

Each rung monetises worse and each is safe. And unlike a feed, the bottom rung is genuinely fine — which is worth saying, because it means the correct engineering trade here is to fail cheaply rather than to build elaborate fallbacks.

The one thing that must never degrade is the budget check. Serving an ad for an advertiser whose budget is exhausted is a billing problem and a trust problem, and it cannot be recovered by a later correction the way a bad ranking can.

Next: the whole thing, as an interview.