Reach Is the Cost Function

In one line: the harm a violating post causes is roughly proportional to how many people see it, so the thing to optimise is not accuracy but how much exposure you prevent.

The reframing

Most designs treat moderation as classification: given a post, decide whether it violates. Get the classifier right, apply the decision, done.

That framing loses the two things that actually determine whether the system works.

Time matters, and it matters unequally. A violating post removed after ten seconds and the same post removed after ten hours are the same decision with wildly different outcomes. If it was spreading, the second one caused several orders of magnitude more harm.

Not all posts are equal — and it is not about severity alone. The overwhelming majority of posts on any platform are seen by almost nobody. A small fraction reach enormous audiences. So a moderation system that is uniformly good is spending most of its capacity on content that would have harmed nobody either way.

The consequence for the design

If harm is severity times exposure, then the quantity to minimise is violating impressions — views of content that should not have been shown — rather than the count of undetected violations.

Three things follow, and each changes the architecture.

The review queue is prioritised by predicted reach, not by score. A post scoring 0.6 on a page with two million followers deserves attention before one scoring 0.9 from an account with four. Ordering the queue by confidence is the intuitive choice and it is wrong.

Speed is worth more than accuracy on the fast-moving tail. For content that is spreading, an imperfect decision now beats a perfect one in an hour. For content that will never spread, the reverse — you can afford to wait and be right.

Re-evaluation is a first-class operation. A post approved when it had nine views should be re-examined when it hits ninety thousand, because the cost of being wrong just changed by four orders of magnitude. Most designs treat the decision as final; it should be a function of current reach.

Predicting reach

Which requires estimating how far something will spread — a prediction problem sitting alongside the policy one.

The signals are available and mostly cheap: the author's follower count and typical engagement, early velocity in the first minutes, the surface it was posted to, whether the ranking system is already amplifying it, and historical spread for similar content.

Two design points worth making.

Early velocity is the strongest signal, and it arrives late. You cannot know at upload time what will spread. So the architecture needs a second look: an initial decision at upload, and a trigger that re-queues content whose velocity crosses a threshold.

The recommender is upstream of the harm. On a platform where distribution is algorithmic, the ranking system decides reach — which means moderation and ranking are coupled. The cheapest intervention is often not to remove the content but to stop amplifying it, and that requires the two systems to talk.

What this does not mean

Two caveats, because the reach framing can be over-applied.

Some harms are not proportional to reach. Content targeting a specific individual — harassment, doxxing, intimate imagery — harms that person regardless of whether anyone else saw it. Child safety content is illegal at any audience. For these categories the priority is severity alone, and reach weighting is inappropriate.

So the priority function is severity times reach for the categories where harm scales with exposure, and severity alone for the categories where it does not. Saying that distinction out loud is worth doing; a system that reach-weights everything will deprioritise exactly the cases with a specific victim.

Low-reach content still matters to the person who posted it. Removing something wrongly is an error whether or not anyone saw it, and a system that only pays attention to popular content will be sloppy everywhere else — where most users actually live.

The two errors, once more

The familiar asymmetry, in its moderation form.

A false negative is violating content left up, and its cost scales with reach. Visible, screenshotted, occasionally a news story.

A false positive is legitimate content removed. The user experiences it as censorship, they often cannot tell why, and the appeal path is usually slow. Invisible in aggregate, and corrosive to trust in a way that compounds.

Same structure as over-refusal in a guardrail system and wrongful declines in fraud: one error is counted and the other is not, so the pressure runs one way. The difference here is that the false-positive cost is not just commercial — it is a legitimacy cost, and a platform seen as arbitrary loses the consent it depends on.

Next: the part that is harder than any model — writing the policy.