Cheat Sheet

Key takeaway

Choosing a metric is choosing which mistake you will make. Establish the cost asymmetry first, let it pick the metric and the threshold, and always be able to say what the metric threw away.

The chain

Classification

Precision  =  TP / (TP + FP)    over your PREDICTIONS
Recall     =  TP / (TP + FN)    over REALITY
F1         =  harmonic mean     assumes they matter equally

Either alone is gameable — quote as a pair, or "recall at 95% precision". The threshold is chosen after training from the cost asymmetry, and changing it is a config change, not a retrain.

Expected cost: FP(t) × cost_fp + FN(t) × cost_fn. Sweep t, take the minimum.

Two thresholds beat one: auto-act, auto-approve, and a human band sized by review capacity.

Imbalance

Metric	Under imbalance
Accuracy	Measures the base rate, not the model
ROC-AUC	Both axes normalised by class size, so skew cancels — random is always 0.5
PR-AUC	Precision keeps predicted-positives in the denominator, so it stays honest

Same model can be ROC-AUC 0.96 and PR-AUC 0.71. Always quote PR-AUC against its base rate — it has no fixed reference point.

Rebalancing in training inflates probabilities. Recalibrate, or every expected-value calculation downstream is wrong.

Ranking

Metric	Assumes
Precision@k	Order within k does not matter
Recall@k	You can enumerate what is relevant
MRR	Exactly one right answer
MAP	Several relevant, relevance binary
NDCG	Graded relevance, and you can label it

Calibration

Calibrated means: of everything scored 0.7, about 70% are positive. Independent of ranking — perfect AUC can have useless probabilities.

Score used for	Calibration
Sorting a list	Irrelevant
A fixed threshold	Matters
Expected value, ensembling, showing confidence	Essential

Measure with a reliability diagram; ECE hides direction and location. Fix post-hoc on held-out data — Platt below ~1,000 points, isotonic above. Recalibrate when the base rate moves. Neural nets are overconfident by default.

Generation

Overlap-with-reference metrics punish correct paraphrase and reward fluent falsehood — regression detectors only, never quality.

If the output is executable, execute it. Report the tail, not the mean — the catastrophic 5% is what causes the incident.

LLM-as-judge

Agreement above 80%, against human-to-human agreement of 81% — accuracy is not the objection. The biases are, because they are systematic and do not average out.

Bias	Magnitude	Fix
Position	Up to 75% for first	Evaluate both orders; flip rate measures noise
Verbosity	Prefers longer	Penalise length in the rubric
Self-preference	10–25% own family	Judge with a different family

Rubric beats model: one dimension at a time, comparison over absolute scoring, few scale points, reasoning before verdict. Never let a judge grade a loop optimising against it.

RAG

Retrieval is the failure ~73% of the time. Ask "was it in the context?" before touching the prompt.

Low metric	Fix
Context recall	Chunking, embeddings, query rewriting
Context precision	Reranker, reduce k
Faithfulness	Grounding instructions, citations, model
Answer relevance	Query understanding

Raising k raises recall, lowers precision, and dilutes attention so faithfulness can fall too — sweep for the interior optimum. Evaluate ingestion: a chunk that cannot be read alone is invisible to query-time metrics.

Eval sets

Three sets: development (iterate freely), regression (only grows, every shipped bug), held-out (look rarely).

Stratify by difficulty, segment, adversarial — then report per stratum, not averaged back.

Uncertainty ≈ 1 / sqrt(n): 100 → 10%, 1,000 → 3%, 10,000 → 1%.

Assume public benchmarks are contaminated. Schedule a staleness refresh.

Experiments

Halving the detectable effect quadruples the sample. Decide the minimum effect worth acting on first.

Peeking inflates false positives — fix the duration or use a sequential method. Plot effect over time: declining is novelty, rising is primacy. Report effect size with an interval, not a p-value.

Quick decision cues

Asked "which metric" → ask what each mistake costs first
Rare positive class → PR-AUC with its base rate; name the ROC trap
Model "not good enough" → sweep the threshold before retraining
Score multiplied by anything → calibration is essential
End-to-end RAG quality poor → check retrieval recall first
Ranking quality poor → check retrieval recall first
Generated text → check parse and citations before anything clever
Judge says the new system wins → swap the order and re-run
Offline win → expect it to shrink; interleave before A/B
Dashboard went green early → peeking; check sample ratio and the time series