The Confusion Matrix and the Threshold

In one line: a model produces a score, a threshold turns that score into a decision, and the threshold is chosen by you rather than learned — which makes it the cheapest lever in the whole system.

The four outcomes

Two metrics fall out, and the way to remember them is by what sits in the denominator:

Precision  =  TP / (TP + FP)      of what we FLAGGED, how much was right
Recall     =  TP / (TP + FN)      of what EXISTS, how much did we catch

Precision is about the cost of acting. Recall is about the cost of missing. That mapping back to the previous lesson's cost asymmetry is the whole reason both exist.

A sentence worth having ready: precision is measured over your predictions, recall over reality. Candidates who confuse them under pressure usually have not anchored on the denominator.

They trade against each other, always

Both are computed at a threshold, and moving the threshold moves both in opposite directions.

The two extreme rows are worth stating because they show the metrics can be gamed individually. Perfect recall is free — flag everything. Near-perfect precision is usually free too — flag only the single most obvious case. Either metric alone is meaningless, which is why they are always quoted as a pair, or at a fixed value of the other.

That last form is the one used in practice and it is worth adopting: "recall at 95% precision" is a real number that cannot be gamed, where "recall" on its own can be.

The threshold is not part of the model

This is the point that separates answers, and it is genuinely under-appreciated.

Training produces a scoring function. The threshold that converts a score into a decision is chosen afterwards, from the score distribution and the cost asymmetry, and it can be changed in production without retraining anything.

Two consequences follow, and both are good things to volunteer.

A "bad" model may be a badly thresholded one. Before retraining, sweep the threshold. It is minutes of work against weeks, and it frequently recovers most of the gap.

The threshold should be revisited when the business changes. If chargeback costs rise, the optimal threshold moves — with no change to the model at all. Designing the threshold as a configurable value rather than a constant in the code is a small decision that pays repeatedly.

Choosing it properly

The defensible method is expected cost. Assign a cost to each error type and pick the threshold that minimises the total:

Expected cost(t)  =  FP(t) x cost_fp  +  FN(t) x cost_fn

Sweep t, compute, take the minimum. This is more defensible than any rule of thumb because it makes the assumption explicit — and if someone disagrees with the answer, they have to disagree with a cost, which is a productive argument to have.

Two other methods that come up:

Fix one metric, maximise the other. "We need 99% precision for auto-blocking; what recall does that give?" Common wherever a hard constraint exists — usually legal, regulatory, or a support-capacity limit.

Fix the volume. "The review team can handle 500 cases a day, so set the threshold at whatever flags 500." Unglamorous and extremely common, and it reframes the model's job as ranking rather than classification.

Where F1 fits, and where it does not

F1 is the harmonic mean of precision and recall:

F1  =  2 x (precision x recall) / (precision + recall)

The harmonic mean is used rather than the arithmetic one because it punishes imbalance. A model at 100% precision and 1% recall has an arithmetic mean of 50.5% and an F1 of about 2% — the harmonic mean refuses to be fooled by one strong number.

That property makes it a reasonable single summary when you genuinely have no cost information. It remains a claim that both errors matter equally, and when you do know the asymmetry, the weighted form lets you state it rather than hiding it.

Next: what happens to all of this when positives are rare.