Reports and Proactive Detection

In one line: reports find what your models cannot and are used to attack people, so the reporter matters as much as the report.

Two detection paths

Proactive — the system finds it. Scales without bound, catches things before anyone sees them, and only finds what it was trained to find.

Reactive — someone reports it. Catches novelty, context and harms that require knowing the target. Depends on someone seeing the content first, which means the harm has already partly occurred.

The two are complements with different coverage, and neither substitutes for the other.

What reports catch that models cannot

Three categories, and they are the ones where the missing information is not in the content.

Harms requiring knowledge of the target. Impersonation, doxxing, revenge imagery, harassment referencing something private. The post can look innocuous; only the victim knows what it means.

Context the system does not have. A photo taken without consent. A quote fabricated but plausible. An in-joke that is actually a coded threat within a community.

Genuinely new patterns. A novel scam format, a new evasion technique, a coded term coined last week. The model has no representation for it; users notice immediately.

That last one makes the report stream a novelty detector, and it is worth treating as such — a sudden cluster of reports on a pattern nothing flags is one of the earliest signals of an emerging attack.

Reports are a low-precision signal

The countervailing fact.

Most reports are wrong, not maliciously but because people report what they dislike. Disagreement, dislike of the person, misunderstanding, and content that is unpleasant without violating anything all produce reports.

So a report is evidence that someone objected, not that a policy was violated. Treating report volume as ground truth builds a system that enforces majority preference — which is a different product from one that enforces a policy, and usually not the one intended.

The design consequence: reports are a prioritisation signal feeding into the queue, not an action trigger. Something reported many times moves up; it does not get removed for having been reported.

The reporter is a feature

The most useful idea here, and it is frequently missed.

Reporters have history. Some report accurately and their reports overwhelmingly result in enforcement. Others report constantly with almost nothing upheld.

So maintain a reporter reliability score — the historical rate at which this person's reports are upheld — and weight accordingly. A report from someone with a strong track record is worth many from accounts with none.

This does several things at once. It suppresses coordinated false reporting, because a brigade's accounts have no history. It surfaces genuine issues faster, because reliable reporters are effectively volunteer moderators. And it is cheap: the data is a by-product of resolving reports.

Two refinements worth adding. Reliability is per category — someone excellent at spotting spam may be unreliable on hate speech. And it needs a cold-start path, or a new user's first legitimate report is ignored; weight new reporters at a neutral prior rather than zero.

Brigading

Coordinated false reporting, and it is common enough to design against explicitly.

The signals are structural rather than content-based: a burst of reports on one target in a short window, from accounts with little history, that are connected — created around the same time, sharing devices or networks, or already linked in the follow graph.

Three responses. Weight by reliability, which handles most of it automatically. Detect the coordination and treat a brigade as its own violation rather than only discounting its reports. And protect targets that attract organised reporting — a repeatedly-brigaded account should get human review rather than automated action, because the automation is the thing being attacked.

That last point generalises: anywhere an automated system can be triggered by third parties, the trigger is an attack surface.

Making reporting work

Four practical points, all of which affect signal quality.

Ask for a reason. A report tagged with a policy category is far more useful than an undifferentiated flag — it routes correctly and it tells you what the reporter thought they saw.

Do not over-ask. Every additional field reduces completion, and the people most affected are often the least willing to fill in a form. There is a real trade between signal richness and coverage.

Close the loop. Tell the reporter what happened. Costly at scale and it is what keeps reliable reporters reporting — and it lets someone correct a misunderstanding rather than concluding the platform ignores them.

Make it easy to report the right thing. A reporting flow that only offers "spam" and "abuse" produces reports labelled spam and abuse regardless of what was actually wrong.

Coverage, and what it hides

The metric that matters here, and it is uncomfortable.

Proactive rate — the fraction of enforced content found by the system before anyone reported it — is the standard measure, and it is reported publicly by large platforms.

Its weakness: a high proactive rate is achievable by being very good at one easy category, while a hard category is entirely report-driven. Report it per policy area, or the aggregate flatters the system on exactly the categories where proactive detection is weakest.

And a rising report rate for a category is a signal that proactive detection there is failing, not that the category got worse. Watching the two together is what makes either interpretable.

Next: the queue, and the people working it.