The Enforcement Ladder

In one line: removing and doing nothing are the two ends of a range, and the useful design lives in between.

Why binary fails here too

A remove-or-keep decision forces every case into one of two outcomes, which is wrong for two independent reasons.

Confidence varies. A post the model scores 0.95 and one it scores 0.55 should not receive the same treatment, and a threshold pretends they do.

Severity varies. Content that is illegal, content that is harmful, and content that is merely unpleasant deserve different responses. Collapsing them into one action means either under-enforcing the first or over-enforcing the third.

Two dimensions, and a binary decision represents neither.

The rungs

Meta's public framing — remove, reduce, inform — is the standard shape, and the full ladder has more steps.

Reduction is the interesting rung

Reducing distribution — showing something less, or not recommending it — is the option most designs never draw, and it resolves the two hardest cases.

Borderline content that does not violate. Sexually suggestive imagery that is not nudity, profanity, gore that is not gratuitous, engagement-bait. Removing it is over-enforcement; ignoring it degrades the experience. Reducing its reach is proportionate.

Content you are unsure about. Where the model scores in the middle, reduction is a hedge: if it is violating, the harm is bounded; if it is not, the cost is some lost reach rather than a wrongful removal.

That second use is the important one architecturally. Reduction lets uncertainty map onto a proportionate action instead of a coin flip — the same structural move as a step-up challenge in fraud or an escalation band in a guardrail cascade, arriving in a third domain.

It is also cheap. On a platform where distribution is algorithmic, reducing reach is a ranking adjustment rather than an enforcement action, which is why the moderation system and the recommender have to be coupled.

What reduction costs

Being honest about it is what separates a designed answer from an enthusiastic one.

It is invisible to the user. Someone whose post was reduced sees no notification, no reason, and no appeal path — because from their side nothing happened. That is exactly the complaint behind claims of shadow-banning, and the complaint is structurally fair: an enforcement action was taken and the affected person was not told.

It is hard to audit. A removal is a discrete, countable event. Reduced reach is a counterfactual — what would this have reached otherwise? — so it does not appear cleanly in transparency reporting, and its effects are difficult for anyone outside the company to verify.

It is easy to over-use. Because it feels less severe, teams reach for it in cases where they have not done the work to decide whether something violates. A reduction applied to avoid making a policy decision is a policy decision made quietly.

Account-level actions

The rung that changes the calculus, because it addresses the actor rather than the artifact.

Removing a post from a coordinated operation accomplishes little; they post again in seconds. Restricting or removing the account raises the cost meaningfully, because reputation and audience are the expensive things to rebuild.

The standard mechanism is a strike system: violations accumulate, with severity weighting, and thresholds trigger escalating restrictions that decay over time.

Three design points:

Severity-weighted, not counted. Three minor infractions and one severe one are different situations.

Decaying. A violation from two years ago should not carry the weight of one from last week, or every long-lived account eventually accumulates a ban.

Appealable at the account level. An account restriction is a much larger sanction than a post removal, and it deserves a proportionally more careful process — which most systems do not provide.

Mapping score to action

Putting the two dimensions together, and this is the answer to "what do you do with the model's output".

                    low confidence    medium         high
  low severity      nothing           label          reduce
  medium severity   reduce            remove         remove
  high severity     human review      remove         remove + account

The bottom-left cell is the one worth explaining. High severity with low confidence goes to human review, not to an automatic action in either direction — because both errors are expensive there and a human is worth the cost.

And the top-right is worth noting too: high confidence on low-severity content gets reduction rather than removal, because being confident that something is mildly unpleasant does not justify taking it down.

Next: the fastest and most certain detection you have.