Calibration
In one line: a model can rank perfectly and still be badly wrong about how confident it is, and that gap only breaks things once you start doing arithmetic with the score.
Ranking and probability are different jobs
A calibrated model is one whose confidence matches its accuracy: of everything it scores 0.7, about 70% should be positive.
A model can rank flawlessly and be badly calibrated. If it outputs 0.99 for every positive and 0.98 for every negative, it separates the classes perfectly — its AUC is 1.0 — and its probabilities are nonsense.
That is fine for some uses and fatal for others.
The third branch is where it becomes non-negotiable. A fraud system computing expected loss as probability × transaction amount is doing arithmetic with the score. If the probability is inflated by a factor of three, every expected-loss figure is wrong by a factor of three, and the whole decision policy is built on it.
The same applies to combining models — an ensemble averaging two miscalibrated scores is averaging two different scales — and to showing a confidence to a user, where "85% confident" is a claim you are making on the model's behalf.
Modern models tend to be overconfident
Worth knowing as a prior: neural network classifiers are commonly overconfident, predicting probabilities that overestimate how often they are actually right.
So the default assumption should be that a deep model's raw scores are not probabilities until you have checked. And as the previous lesson noted, class weighting or resampling makes this systematically worse by distorting the effective base rate.
Measuring it
A reliability diagram is the visual test. Bin predictions by confidence, and for each bin plot mean confidence on one axis against observed accuracy on the other. A perfectly calibrated model traces the diagonal. Bars below the diagonal mean overconfidence.
Expected calibration error compresses that into one number: the average absolute gap between confidence and accuracy across bins, weighted by how many predictions fall in each.
Its weakness is the usual one for a summary statistic. A model overconfident at the high end and underconfident at the low end can average out to a small ECE while being badly wrong in both places — and the high-confidence region is usually the one you act on. Read the diagram.
Fixing it
Calibration is a post-processing step fitted on a held-out set, and it does not change the ranking at all — both standard methods are monotonic, so AUC is untouched.
| Platt scaling | Isotonic regression | |
|---|---|---|
| Fits | A logistic curve on the scores | Any monotonic step function |
| Assumes | A specific sigmoid shape | Only that the mapping is monotonic |
| Data needed | Works with little | More — it overfits on small sets |
| Use when | The calibration set is small | You have roughly a thousand points or more |
The practical rule: isotonic regression once you have around a thousand calibration points or more, Platt scaling below that. Isotonic is more flexible and that flexibility is what overfits when data is scarce.
Two operational points that matter more than the choice between them.
Calibrate on held-out data. Fitting the calibrator on the training set gives you a calibrator that is itself overfitted, which produces confident nonsense — the exact failure you were trying to fix.
Recalibrate when the base rate moves. A calibrator is fitted against a particular positive rate. When that rate shifts — seasonally, or after a fraud wave, or because an upstream filter changed — the calibration drifts even though the model is unchanged. This is a monitoring item, not a one-off task.
When you can skip it
Say this too, because over-applying it is also a mistake. If the score is only ever used to sort a list — a recommendation carousel, a search ranking — calibration is irrelevant, and spending effort on it is spending effort on a number nobody reads.
The test is simple: is the score's magnitude used anywhere, or only its order?
Key takeaway
Calibration is whether the score means what it says, and it is independent of ranking quality — a model with perfect AUC can have useless probabilities. It matters when you threshold at a fixed value, and it is essential when you do arithmetic with the score, such as expected value or ensembling. Measure with a reliability diagram and read it rather than trusting ECE alone. Fix it post-hoc on held-out data with Platt scaling or, above roughly a thousand points, isotonic regression — and recalibrate when the base rate moves, because a shifted base rate breaks calibration without touching the model.
Next: what to do when there is no single correct answer to compare against.