Class Imbalance Breaks the Obvious Metrics
In one line: when the positive class is rare, accuracy is trivially high and ROC-AUC is misleadingly high — and knowing why the second one happens is a genuine level signal.
The accuracy paradox
Fraud is roughly 0.1% of transactions. A model that predicts "not fraud" for everything achieves:
Accuracy = 99.9% Precision = undefined (it never predicts positive) Recall = 0%
99.9% accuracy and it has learned nothing. This example is well known, and the general form is the part worth internalising: accuracy is dominated by the majority class, so it measures the base rate rather than the model.
The rule of thumb: if the positive class is under about 10% of your data, accuracy is not a metric, it is a description of your dataset.
ROC-AUC is the subtler trap
Most candidates know the accuracy problem. Far fewer can explain why ROC-AUC also misleads under imbalance, and explaining the mechanism is what distinguishes a strong answer.
The ROC curve plots true positive rate against false positive rate:
TPR = TP / (TP + FN) normalised by the POSITIVE class size FPR = FP / (FP + TN) normalised by the NEGATIVE class size
Look at the denominators. Both are normalised by their own class size, so the class imbalance cancels out. That is exactly why a random classifier scores 0.5 on ROC-AUC no matter how skewed the data is — the metric was designed to be insensitive to the base rate.
That insensitivity is a feature in some contexts and a serious problem in this one.
Work the arithmetic, because it is what makes the point land. With a million transactions and 1,000 frauds:
Flag 10,000 legitimate transactions as fraud. FPR = 10,000 / 999,000 = 1% -- looks excellent But among everything you flagged: Precision = 1,000 / 11,000 = 9% -- 91% of flags are wrong
A 1% false positive rate sounds like a tightly controlled system. It means nine out of ten alerts are false, and the review team is drowning. ROC saw a small number; the reviewers experienced a large one, and the difference is entirely the base rate.
The precision-recall curve is the honest one
Precision has the predicted-positive count in its denominator rather than a class size, so it does not cancel the imbalance. When positives are rare, a random classifier gets low precision, and the metric reflects reality.
The empirical gap is usually large. The same model can score around 0.96 on ROC-AUC and 0.71 on PR-AUC — the first suggests a nearly solved problem, the second says the positive class is not being captured well.
| ROC-AUC | PR-AUC | |
|---|---|---|
| Axes | TPR against FPR | Precision against recall |
| Sensitive to base rate | No — it cancels out | |
| Random baseline | Always 0.5 | The positive base rate |
| Rewards | Ranking positives above negatives overall | Getting the top of the ranking right |
| Use when | Classes are roughly balanced, or both errors matter symmetrically | The positive class is rare and is what you care about |
The random-baseline row is the practically useful one. A PR-AUC of 0.30 sounds poor until you note the positive class is 1% of the data, which makes the random baseline 0.01 — so the model is thirty times better than chance. Always quote PR-AUC against its base rate, because unlike ROC-AUC it has no fixed reference point.
What to actually do
Reporting the right metric does not fix the imbalance, and the interviewer will follow up. The moves, roughly in order of how much they cost:
| Approach | What it does | The catch |
|---|---|---|
| Report PR-AUC and precision at fixed recall | Nothing to the model — makes the truth visible | It only measures; it does not improve |
| Threshold tuning | Moves the operating point | Bounded by the model's actual ranking ability |
| Class weighting in the loss | Makes positives count more during training | Distorts the output probabilities — recalibrate |
| Resampling | Rebalances the training distribution | Undersampling loses data; oversampling risks memorisation |
| Cascade | Cheap stage removes obvious negatives, expensive stage sees a balanced set | The cheap stage's recall becomes a hard ceiling |
The last row connects to the serving cascade from the estimation chapter and is worth naming as the same idea. A first stage that removes 99% of obvious negatives hands the expensive model a far more balanced problem — which improves both cost and learnability at once.
The interview move
When a problem has a rare positive class, say so before being asked and name what it breaks:
"Fraud is well under 1% of transactions, so accuracy is meaningless here — predicting 'never fraud' scores 99.9%. ROC-AUC will also flatter us, because both its axes are normalised by class size so the imbalance cancels; a 1% false positive rate against a million transactions is ten thousand false alerts. I'd report PR-AUC alongside its base rate, and precision at whatever recall the review team can absorb."
That is four sentences and it demonstrates the whole lesson.
Key takeaway
Under imbalance, accuracy measures the base rate rather than the model. ROC-AUC misleads for a specific reason worth being able to state: both its axes are normalised by class size, so the imbalance cancels and a huge negative class absorbs enormous false-positive counts without moving FPR. Precision keeps the predicted-positive count in its denominator, so PR-AUC stays honest — but it has no fixed baseline, so always quote it against the base rate. And if you rebalance during training, recalibrate, or every expected-value calculation downstream is wrong.
Next: the metrics for problems where the output is a ranked list.