Cheat Sheet
Key takeaway
One property drives the whole round: the central component's behaviour is learned, not specified. So correctness is a distribution, the data pipeline is architecture, the system writes its own next training set, and failure is silent.
The spine
Time for 45 minutes: 5 scope · 5 metrics · 10 data and design · 15 deep dive · 10 evaluation and failure.
The translation chain
business goal -> ML objective -> measurable proxy -> training loss
| | |
loses everyone loses what you loses what the
you have no cannot observe loss cannot express
data on
The proxy is where designs fail: chosen for being measurable, then optimised until it comes apart from the goal. A stronger model makes a bad proxy worse, not better.
Objective vs guardrail
| Add to the objective | Add as a guardrail | |
|---|---|---|
| Changes | What the model learns | What you will ship |
| Requires | A weight you must justify | A threshold you must agree |
| Use when | The concern is continuous and central | The concern is a floor you must not cross |
Name two or three guardrails unprompted. Anchor each threshold to a known-acceptable state or to a decision.
Key terms
| Term | One line |
|---|---|
| Proxy metric | A measurable stand-in for what you actually want |
| Guardrail metric | Not optimised; blocks a launch if it regresses |
| Point-in-time correctness | Every training feature reconstructed as of the decision moment |
| Feature skew | Same feature, different computation in training and serving — a bug |
| Distribution skew | Same computation, different population — a condition |
| Exposure bias | Logs contain only items the current policy chose to show |
| Inter-annotator agreement | Human ceiling on achievable model accuracy |
| Interleaving | Blend two rankings for one user; detects a difference cheaply |
| Holdback | A slice never given the new system, kept for months |
| Cascade | Cheap model shrinks the set, expensive model ranks the survivors |
Labels
Five sources: explicit signal (tiny), implicit behaviour (huge, position-biased), human annotation (costly, has a ceiling), weak/programmatic (fast, encodes blind spots), downstream outcome (honest, late).
Train on the fast proxy; evaluate against the slow truth. Log the model version with every decision, because the truth lands after you have already shipped something else.
Cold start is asymmetric: a new item has content, so score it from content. A new user has nothing, so it is a policy problem — popularity, free context, and an exploration budget.
Leakage checklist
- An aggregate computed over the full dataset
- A field back-filled by a later process
- An ID correlated with the label
- A join against a current dimension table
- The same entity in train and test
The tell: an offline metric that is suspiciously good. Ask of the top feature — could this have been known at decision time?
The evaluation ladder
| Rung | Buys | Costs |
|---|---|---|
| Offline on logs | Instant; runs on every change | Biased toward the old policy |
| Counterfactual replay | Estimates a new policy from old data | High variance; needs logged propensities |
| Interleaving | Very sensitive, little traffic | Ranking problems only |
| A/B test | The real answer | Slow and expensive |
| Long-term holdback | Catches slow harms | Months, and a withheld population |
Offline evaluation is a filter that protects scarce experiment capacity, never proof of impact. Check sample ratio mismatch before trusting any experiment.
Serving levers
Retrieval optimises recall; ranking optimises precision. An item dropped in retrieval is gone for good. Budget the whole request, not just the model, and keep headroom for p99.
Drift and monitoring
| Drift | What moved | Needs labels |
|---|---|---|
| Data | The input distribution | No |
| Concept | The input-to-target relationship | Yes |
| Prediction | The output distribution | No |
Concept drift is the one that invalidates the model and the one you cannot see quickly — so watch input and prediction distributions as the leading indicator.
Slice everything by segment — the aggregate can improve while new users collapse. On-call needs a tested non-ML fallback and a model that tolerates an absent feature.
Quick decision cues
- Brief is vague → define "better" before drawing anything
- Asked for a model first → give the objective and budget first
- Metric could be gamed → name the exploit, add a guardrail
- Offline win → predict it shrinks online, and say why
- Ranking problem, low traffic → interleaving before A/B
- Model cannot score the catalogue → derive the cascade from the arithmetic
- Feature is great offline → ask whether it exists at decision time
- New item vs new user → content features vs popularity plus exploration
- Labels arrive late → train on the proxy, log the model version
- Asked "what breaks?" → silent degradation, drift, feedback loops
- Asked "how do you know?" → the ladder, not the offline number