Over-Refusal, the Failure You Never Hear About

In one line: every guardrail has a false-positive rate, almost nobody measures theirs, and the resulting damage never appears in any incident report.

Two errors, two very different feedback loops

A false negative is loud. Harmful output reaches a user, somebody screenshots it, it becomes a ticket or a news story, and the team fixes it that week.

A false positive is silent. A nurse asks a legitimate clinical question and gets refused. A security engineer asks about an attack technique and gets refused. A novelist asks for a scene involving conflict and gets refused. Each rephrases once, gets refused again, and stops — quietly forming the view that the product is unreliable for their actual work.

The asymmetry is structural, and it biases every safety team the same direction: the pressure you feel comes from one side only.

The arithmetic

At a million requests a day, a guard with a 4% false-positive rate refuses 40,000 legitimate requests. At 15% it refuses 150,000. Both numbers are entirely plausible for a threshold tuned only against harm.

Now compare against the thing you bought. If genuinely harmful requests are 0.1% of traffic — a reasonable order of magnitude for a mainstream product — that is 1,000 requests a day worth catching. Trading 40,000 wrong refusals for 1,000 right blocks is a ratio a product owner would never approve if anyone put it on a slide.

Nobody puts it on a slide, because the safety dashboard counts blocks and calls them catches.

How it is actually measured

You cannot measure over-refusal on ordinary traffic, because ordinary traffic is overwhelmingly benign and obviously so — you would score 99% and learn nothing. You need prompts that are safe but look unsafe.

XSTest (Röttger et al., 2024) introduced the term exaggerated safety and built a deliberately small hand-crafted suite: 250 safe prompts that superficially resemble unsafe ones, paired with 200 genuinely unsafe contrast prompts. The pairing is the clever part — it stops a system from scoring well by simply refusing less often, because the contrast set catches that immediately.

OR-Bench (Cui et al., ICML 2025) scaled the same idea: roughly 80,000 over-refusal prompts across ten rejection categories, a ~1,000-prompt hard subset, and 600 genuinely toxic prompts to prevent gaming, evaluated across 32 models.

The construction principle transfers directly to your own product, and this is the part to say in an interview: build a benign set that is adversarial to your guard, and pair every benign prompt with a harmful twin. Without the twins, "improving" over-refusal is indistinguishable from weakening the guard.

Where the false positives cluster

They are not uniform, which is what makes them manageable — and also what makes them a fairness problem.

Professional domains. Medicine, security research, law, journalism. The vocabulary of the job overlaps the vocabulary of the harm.

Fiction and history. Conflict, crime and atrocity are the substance of both, and a classifier trained on intent struggles with a frame it cannot see.

Scary-sounding but harmless language. "Kill a process", "inject a dependency", "execute a query", "hijack the scroll event".

Non-English and code-switched text. Guard models are trained overwhelmingly on English. False-positive rates are consistently worse elsewhere, so an untuned guard delivers a materially worse product to your non-English users — a fairness issue as much as a quality one, and one that only shows up if your eval set is multilingual.

Reclaimed and in-group language. Terms that are slurs from outside a community and ordinary usage within it.

Setting the threshold on purpose

The threshold is a product decision that most teams make by accident.

Start from the two costs. What does one harmful output cost — reputationally, legally, in this domain? What does one wrong refusal cost? On a children's education product the first dominates and you accept a high false-positive rate. On a security research assistant the second dominates, because a tool that refuses half the questions in its own subject is worthless.

Then set per-category thresholds, not one global one. The tolerable false-positive rate for content involving minors is near zero; the tolerable rate for mild profanity is much higher. A single threshold across a taxonomy is a category error — it forces the least tolerant category to set the level for all of them.

And use the uncertain band. The two-threshold cascade exists precisely so borderline cases get judgement instead of a coin flip.

Make the silent failure countable

Since users will not tell you, instrument it.

Ask. A one-click "this shouldn't have been blocked" on every refusal. Low response rate, high signal, and it doubles as your appeals intake.

Watch the behaviour after a refusal. A user who rephrases and retries twice, then abandons the session, has told you something without filling in a form. Refusal-then-abandonment is the closest thing to a direct over-refusal metric that live traffic offers.

Sample and review. A random sample of blocks, labelled by humans weekly. Slow, expensive, and the only ground truth you will have.

Track the rate by segment. By language, by user cohort, by surface. A global rate of 3% can hide 20% in one language.

Next: running the whole layer — policy, red teaming, review and incidents.