Interview Walkthrough: Designing the Evaluation for a Moderation System

The prompt: "We're building content moderation for a social platform. How do you know if it's working?"

Step 0 — Refuse the single number

Candidate: "Before I pick a metric I need to know what we're moderating, because the cost asymmetry inverts between categories. For child safety material, a false negative is catastrophic and a false positive is a minor inconvenience. For political speech, a false positive is a censorship incident and a false negative is a piece of content people argue about. Same system, opposite objectives — so I don't think there's one metric here."

Interviewer: "Say we have three tiers: illegal content, policy violations like harassment, and borderline spam."

That is the answer to design against, and it lets everything else be concrete.

1. Cost asymmetry per tier

Candidate: "So three thresholds, not one — and probably three models eventually, because a single model forced to serve all three objectives will be mediocre at each. For now, one model with per-category thresholds is the cheaper starting point."

2. The metric, and the trap

Candidate: "Violating content is maybe 0.5% of posts, so I want to name what that breaks before I quote any number. Accuracy is useless — 'approve everything' scores 99.5%. And ROC-AUC will flatter us, because both its axes are normalised by class size so the imbalance cancels out."

Then make it concrete, because the arithmetic is what lands:

10,000,000 posts/day, 0.5% violating  =  50,000 violations

A 1% false positive rate sounds tight. It means:
  1% of 9,950,000  =  99,500 legitimate posts wrongly flagged
  against 50,000 real violations
  -> precision under 34%, two-thirds of flags are wrong

Candidate: "So I'd report PR-AUC against its base rate, and precision at whatever recall we need per tier. And the real operating metric is precision at the recall the policy demands — for illegal content that recall is close to non-negotiable, so the question becomes what precision we can achieve at it."

3. Threshold, and the band

Candidate: "I wouldn't use one threshold. Two, giving three outcomes."

Candidate: "The band width is set by review capacity, not by the model. If the team can handle 20,000 items a day, that's the budget. And the band is where the training data comes from — those are the hardest cases and the ones labelling improves the model fastest on."

Interviewer: "How do you set the auto-remove threshold?"

Expected cost. A wrongful removal has an appeal cost and a reputational cost; a missed violation has a harm and legal cost. I'd put rough numbers on both and pick the threshold minimising total expected cost, then sanity-check the resulting volumes. The numbers will be argued about — that's the right argument to have, because it's about values rather than about statistics.

4. Offline evaluation

Candidate: "Stratified, and reported per stratum."

Candidate: "The regression set matters most here. Every successful appeal is a labelled false positive that a human already adjudicated — it's the highest-quality data in the system and it arrives free. And I'd report by language, because moderation quality collapses on lower-resource languages and an aggregate hides it completely."

Interviewer: "Where do the labels come from?"

Human annotators with a rubric, and I'd track inter-annotator agreement — because on harassment two trained reviewers might agree only 70% of the time, and that caps what any model can achieve. If agreement is that low the fix is a better rubric, not a better model. That's the number I'd want before anyone reports model accuracy.

5. Online

Interviewer: "You've improved offline recall by 4%. Ship it?"

Not yet. Offline recall is measured against labelled data, and our labels come from content that was already flagged by the current system — so we mostly cannot see the violations it never surfaced. I'd expect offline to overstate the win.

I'd run an experiment on the primary metric with guardrails on appeal rate and reviewer queue depth, check sample ratio first, and plot the effect over time rather than reading one aggregate — because a moderation change can look good in week one while evaders adapt in week three. And I'd hold a long-term holdback, since the real question is whether harmful content declines over months, not whether flags went up this week.

Interviewer: "What breaks silently?"

Two things. The label distribution drifts as evasion tactics change, which is concept drift — the same features mean something different and I cannot see it until new labels arrive. So I'd watch the score distribution and the appeal rate as leading indicators, both of which need no labels. And calibration: if we rebalanced training to handle the 0.5% base rate, the probabilities are inflated, and any policy multiplying score by severity is then wrong.

Now do it live