The Metric Is the Design Decision
In one line: choosing a metric is choosing which kind of mistake you are willing to make, so it is a product decision wearing a statistical costume.
Two errors, never equally bad
Every classifier makes two kinds of mistake, and the entire subject follows from noticing they are almost never equally expensive.
| System | A false positive costs | A false negative costs | Which is worse |
|---|---|---|---|
| Cancer screening | An unnecessary follow-up test | A missed diagnosis | The false negative, enormously |
| Spam filter | A real email lost in junk | Some spam in the inbox | The false positive |
| Fraud blocking | A legitimate customer declined | A fraudulent charge | It depends on the amount |
| Content moderation | Legitimate speech removed | Harmful content published | Contested, and it varies by category |
The spam and cancer rows point in opposite directions, and no metric is correct for both. This is why "what is the best metric for classification?" has no answer — the question is incomplete until someone says what a mistake costs.
The fraud row is the interesting one, because the answer changes within the system. Declining a legitimate five-dollar purchase is a minor annoyance; declining a legitimate five-thousand-dollar one loses a customer. A single threshold across all transactions is already the wrong design, and noticing that from the metric discussion rather than from the architecture is the point of this chapter.
The chain from cost to architecture
Read that chain in both directions and it explains a lot of the round.
Forwards, it is how a good answer is built: establish the cost asymmetry, pick the metric that expresses it, set the threshold where the costs balance, and then discover the architecture has to support things you had not drawn — a human review queue for the uncertain band, different thresholds per segment, a cheap first stage.
Backwards, it is a diagnostic. A candidate who proposes an architecture with no review queue has implicitly claimed the errors are cheap. If they cannot defend that, the design has an unexamined assumption in it.
There is no default
The instinct to reach for accuracy, or for F1, is worth naming and resisting.
Accuracy answers "what fraction did we get right", which is only meaningful when the classes are balanced and the errors cost the same. Both conditions fail in most real systems, and the next lesson shows how badly.
F1 is the harmonic mean of precision and recall, which encodes a specific opinion: that precision and recall matter equally. That is a strong claim and it is usually false. It is a reasonable default when you genuinely have no information about the cost asymmetry, and it is the wrong answer whenever you do.
Say the cost out loud, in units
The strongest move available in this part of an interview is to convert the asymmetry into a number, even a rough one.
"A false positive here costs us a support contact, maybe four dollars. A false negative is a chargeback plus the goods, call it eighty. So a false negative is roughly twenty times worse, and I want the threshold set where the marginal expected costs balance — not at 0.5."
That does several things at once. It shows the metric follows from the business rather than from habit. It gives the threshold a justification instead of a default. And it opens the door to expected-value decisions rather than classification, which is where the strongest answers end up.
When one number is not enough
Sometimes the honest answer is that a single metric cannot express what you want, and saying so is stronger than forcing one.
The usual resolution is the pattern from the previous chapter: one primary metric you optimise, plus guardrails that block a launch. A moderation system might optimise recall on genuinely harmful content while holding a guardrail on the false-positive rate for a protected category. Neither number alone is the objective; the pair is.
What the rest of the chapter does
The lessons ahead work through the metric families and what each one assumes: the confusion matrix and threshold selection, why class imbalance breaks the obvious choices, the ranking metrics, calibration, evaluating generation where no single right answer exists, and the experiment design that turns a measurement into a decision.
The through-line is the same in every one: a metric is a compression of reality, and the interview question is always what it threw away.
Key takeaway
There is no default metric, because choosing one is choosing which mistake you are willing to make. Establish the cost asymmetry first — ideally in currency — and let it pick the metric, then the threshold, then the parts of the architecture that only exist because errors are expensive. Accuracy assumes balanced classes and equal costs; F1 assumes precision and recall matter equally. Both are claims about your product, and both are usually wrong.
Next: the confusion matrix, and why the threshold is not a model property.