Evaluation by Cost, Not by AUC

In one line: AUC integrates over operating points you will never use, and this system runs at exactly one.

Why AUC is the wrong headline

AUC averages performance across every possible threshold. Your system uses one — or two, with a challenge band — chosen from a cost matrix and a review capacity.

So a model with better AUC can be worse at your operating point, because the improvement lives in a region of the curve you never occupy. This is not a subtlety; on heavily imbalanced problems it is common.

There is a second problem. AUC is computed over the full distribution, which is overwhelmingly obvious legitimate traffic. A model gets most of its AUC from separating trivial cases, and the differences that matter are in a thin band of genuinely ambiguous transactions that barely register in the aggregate.

What to report instead

Four, and the first is the one that should lead.

Expected cost at the operating point. Combine the confusion matrix with the cost matrix and produce a number in currency. This is the only metric that speaks the business's language, and it makes the trade explicit rather than implied.

expected cost = (missed fraud x average loss)
              + (wrong declines x lost customer value)
              + (challenges x abandonment cost)
              + (reviews x analyst cost)

Precision and recall at the operating point, not averaged. If review capacity binds, precision at the fixed review volume is the metric — a better model reviews better-chosen cases, not more of them.

PR-AUC if you want a single curve-based number, since it is far more informative than ROC-AUC under this imbalance. Useful for tracking, still not the decision.

Value-weighted recall. The fraction of fraud value caught, not the fraction of fraud transactions. Fraud concentrates in the tail, so a model can catch most incidents and miss most money — and only the value-weighted version notices.

The experiment

An ordinary A/B is available here and has three complications.

The outcome arrives late. A two-week experiment produces two weeks of decisions and almost no matured labels. So you measure leading indicators immediately — decline rate, challenge rate, score distribution, approval rate by segment — and the actual fraud outcome months later. Which means the decision to ship is made on proxies, and the confirmation comes long after.

The population is shared with an adversary. Attackers do not respect your randomisation. If treatment is more permissive, fraud flows toward whichever arm is easier, so the arms interfere through the attacker's own routing. Treatment absorbs fraud the control would otherwise have taken, which flatters control and understates the difference.

Interventions change future behaviour. A declined customer may not return, so the treatment arm's population diverges from control over the experiment's life. That is a genuine carryover effect, not noise.

The practical response: run longer than feels necessary, watch leading indicators for the ship decision and matured labels for the verdict, and prefer randomising by entity rather than by transaction so a customer has a consistent experience and carryover is at least measurable.

Backtesting, and its ceiling

Replay historical traffic through the candidate model and see what it would have decided. Fast, cheap, and it is the right first filter.

Two limits to state, because an interviewer will probe them.

It cannot evaluate declines. For transactions the old system declined, there is no outcome, so a new model that would have approved them scores against nothing. Backtesting can tell you a model catches more of the fraud you already caught; it cannot tell you what it lets through.

It ignores the adversarial response. The historical attacker was optimising against the old system. A new model changes what attackers do, and the replay holds that fixed.

So backtesting bounds the downside and does not measure the upside. It is a gate, not a verdict — the same conclusion as offline metrics in the ranking chapters, for a related reason.

What else to watch

Three metrics that are not model metrics and matter more than most of them.

Approval rate by segment. A model that improves overall while collapsing approvals for a customer segment is a fairness problem and a revenue problem simultaneously. Watch new customers specifically — they have the least history and are the easiest to decline by accident.

Review queue health. Volume, age, and the analyst overturn rate. A rising overturn rate means the threshold is wrong; a growing queue means the operating point is unsustainable regardless of what the offline metric says.

Total loss across all channels. The point from the first lesson. Improving one channel while fraud relocates is invisible in that channel's metrics, and only the total shows it.

The reporting trap

Worth ending on, because it causes real incidents.

Recent fraud always looks low, because the chargebacks have not arrived. A model shipped on Monday looks excellent all week and the regression appears six weeks later.

Two protections. Report fraud rates only over fully matured windows, and mark the immature tail explicitly on every chart. And maintain a maturity curve — the historical shape of how chargebacks accumulate over time — so a partial window can be scaled to an expected final value rather than read at face value.

That second one is cheap and turns an unusable recent window into a noisy but honest estimate, which is what you need to decide anything at all in the meantime.

Next: serving it, and what happens when a dependency fails.