The Label Problem

In one line: you find out about fraud months later, only for transactions you approved, and the ones you declined have no outcome ever.

Three separate defects

They are late. A chargeback can arrive months after the transaction. So today's model is trained on labels from a quarter ago, against an adversary who has changed twice since.

They are incomplete by construction. You observe the outcome of an approved transaction. A declined one has no outcome — it did not happen. Whether it would have been fraud is unknowable, forever.

They are wrong in both directions. Not every chargeback is fraud — some are buyer's remorse or a customer not recognising a merchant name, which the industry calls friendly fraud. And not every fraud is charged back: a victim who never noticed, or a small amount below the effort of disputing, is fraud labelled legitimate.

The missing half

The second defect is the deepest thing in this chapter, and it is worth being precise about.

The model was trained on approved transactions. It is evaluated on approved transactions. It is deployed to decide which transactions get approved.

So the training distribution is produced by the previous version of the model, and the region the model considers risky is systematically absent from its own data. It never learns it was wrong to decline, because a decline generates no evidence.

The self-confirming failure follows: decline a legitimate segment, observe no fraud from that segment — because you observe nothing from it — and conclude the decline was correct. The model's belief is unfalsifiable within its own data.

This is the same structure as the recommendation feedback loop and the ads cold-start trap. Here it is sharpest, because the decision is a hard block rather than a ranking, so the missing region is a clean hole rather than a thin slice.

Rejection inference, and why the naive versions fail

The credit-scoring literature has a name for this and a set of techniques, most of which do not work as advertised.

Treat all declines as fraud. Guaranteed to make the model more conservative every generation, because it confirms whatever it already believed. A self-fulfilling spiral, and the most common approach in practice.

Treat all declines as legitimate. Equally arbitrary, in the other direction.

Impute with the current model. Predict the missing labels using the model that produced the gap. This is circular by construction: the imputed labels agree with the model, so training on them reinforces its existing boundary while appearing to add data.

Recent work is blunt about the effect: biased models score well partly because they are evaluated on a population their own decisions shaped. The measurement and the bias share a cause.

What actually works

Only one thing genuinely closes the hole, and everything else narrows it.

Approve a random sample of what you would have declined. A small slice, deliberately let through, with the loss accepted as the cost of the information. Those transactions produce real labels in the region you cannot otherwise see — and they are the only unbiased data you will ever have about your own decision boundary.

That is expensive and it is the honest answer. The cost is bounded by the sample rate and can be concentrated in the low-exposure part of the distribution: sample more heavily among small-amount declines, where a wrong approval costs little, and barely at all in the high-exposure tail.

Three partial substitutes, all worth having and none sufficient:

The challenge band is free labelling. A step-up that is passed or failed generates a signal on a transaction you were unsure about, without approving it. Not the same as an outcome, and much cheaper than one.

Analyst review supplies fast, expert labels on a sample of declines — a human judging whether the decline was right. Biased by the analyst's own beliefs, and far better than nothing.

Downstream signals are weak labels. A declined customer who successfully completes the same purchase minutes later through another path was probably legitimate. That is inference rather than truth, and it is available at no cost.

Handling the delay

Separate from the hole, and it has its own answers.

Multiple label horizons. Train one model on labels matured to thirty days for accuracy, and one on early signals for responsiveness. Blend by how much the recent window can be trusted.

Fast proxies. Some fraud reveals itself quickly — a failed step-up, an immediate account takeover report, a card reported stolen within hours. These arrive in days rather than months and are strong enough to train on.

Never evaluate on an open window. Last week's fraud rate always looks low because the chargebacks have not arrived. Teams read that as improvement and ship. Report only over windows that have fully matured, and mark the immature tail explicitly.

That last one causes real incidents. A model shipped on Monday looks excellent all week — because its labels have not landed — and the regression appears six weeks later.

Label quality

The friendly-fraud problem deserves its own treatment, because it is a labelling error rather than a modelling one.

A chargeback coded as fraud may be a customer who did not recognise the merchant descriptor, or who is disputing a legitimate purchase to get a refund. Training on that teaches the model to decline ordinary customers.

The mitigations are unglamorous and effective: use the dispute reason code rather than treating all chargebacks alike; hold out cases the merchant successfully represented; and track the friendly-fraud share as its own metric, because it moves with product changes — an unclear billing descriptor generates disputes that look exactly like fraud in your training data.

Next: keeping the system current against someone who is adjusting.