Policy Is the Hard Part

In one line: you cannot build a classifier for a rule nobody can apply consistently, so the policy is the specification and its quality bounds everything downstream.

The gap

"No hate speech" is a principle. It is not a rule.

Applying it requires deciding: is this about a protected characteristic or an individual? Is it an attack, a description, or a report of someone else's attack? Is the slur reclaimed in-group usage? Is it satire? Does the speaker's identity matter? Does the target's?

Each of those is a real decision that changes the answer, and none is settled by the principle. Until they are settled, two trained reviewers looking at the same post will disagree — and if they disagree, no model can do better, because their labels are the training data.

The enforceability test

The check to state, because it makes the abstraction concrete:

Could two trained reviewers, in different countries, applying only the written policy to the same post, reach the same decision?

If not, the policy is a principle wearing a rule's clothes, and it will produce inconsistent labels, an inconsistent model, and enforcement that users correctly perceive as arbitrary.

Inter-reviewer agreement is therefore the ceiling on your model's accuracy. If your reviewers agree 75% of the time on a policy, a model scoring 75% against their labels is at the ceiling and further effort is chasing noise in the guidelines rather than error in the model.

Which produces a genuinely useful conclusion: when agreement is low, the highest-value work is rewriting the policy, not improving the model. That is a surprising thing to say in a machine learning interview and it is usually correct.

What makes a policy enforceable

Four properties, and each one is missing from most first drafts.

Decision criteria in order. Not a description of the harm but an ordered sequence of questions whose answers determine the outcome. Is there a target? Is the target a protected group? Is the statement an attack? Ordered, so reviewers reach the same place by the same route.

Worked examples on both sides of the line. The boundary is defined by cases, not by prose. A policy with ten violating examples and no near-miss examples teaches reviewers to over-enforce, because they have never been shown what a close call that stays up looks like.

Explicit exceptions. Newsworthiness, education, documentation of abuse, counter-speech, artistic context. Every content policy has them and an unstated exception is applied inconsistently by definition.

A stated default. What happens when the policy genuinely does not cover a case. Escalate, or leave up, or remove — but decided in advance, because the alternative is each reviewer inventing their own.

Context that is not in the post

The difficulty that makes this different from ordinary classification.

The same words can be a slur, a quotation of a slur in a news report, reclaimed in-group speech, or a lyric. The same image can be documentation of an atrocity or its celebration. A phrase innocuous in one country is a coded political attack in another.

The signals that resolve this often live outside the post: who posted it, who they are addressing, the thread it sits in, the account's history, the local moment.

Which produces two design consequences.

The classifier needs context features, not just content features. A model scoring the post alone is missing the information the decision depends on, and no amount of training fixes an input that is absent.

Some cases genuinely cannot be automated. Satire, counter-speech and newsworthiness require judgement the model does not have. The right response is to route them to humans rather than to force a decision — which is only possible if the policy names them as routing categories rather than as edge cases.

Policy as versioned data

The operational half, and it mirrors the guardrail chapter's treatment for the same reasons.

Policies change constantly — new harms, new regulation, new markets, a news event that makes a previously-fine phrase a coded attack. If each change requires a code deploy, the team stops making them.

So the policy lives in a store with a version, an owner, decision criteria, worked examples on both sides, and per-region variation. Every enforcement decision is logged with the policy version that produced it — otherwise "why was this removed?" is unanswerable a month later, which is a problem for the user, the appeal, and the regulator simultaneously.

And the examples do triple duty: they are the specification for reviewers, the evaluation set for the model, and its training data. A policy without examples is an opinion, interpreted differently by everyone who reads it.

Where policy and model diverge

One subtlety worth raising.

A model trained on reviewer decisions learns what reviewers actually do, which is not identical to what the policy says. Where reviewers systematically drift — over-enforcing a category because a past incident made them cautious — the model learns the drift and then scales it to everything.

The check is to periodically audit model decisions against the written policy rather than against reviewer labels, using a small expert panel. It is slow and expensive and it is the only way to catch a model that is faithfully reproducing a systematic human error.

Next: the enforcement options between removing and doing nothing.