The Input Guard Cascade
In one line: you cannot afford to run your best safety classifier on every request, so you run cheap checks first and escalate only what they are unsure about.
The budget problem
Suppose you want three things checked before generation starts: an obvious deny-list, a general harm classifier, and a nuanced policy judgement. Run all three serially on every request and the third dominates.
The numbers are unforgiving. A regex pass is well under a millisecond. A small encoder classifier — a BERT-sized model over a few hundred tokens — is single-digit to low-tens of milliseconds on a GPU. An LLM-based judge is a full transformer forward pass; Meta's Llama Guard family lands in the hundreds of milliseconds, and its 1B variant exists precisely because the 8B is too slow for an interactive path.
Add that to a first-token latency you are already fighting for, and a naive perimeter has doubled your time to first token before the model has done anything.
The shape of the answer is a cascade
The same structure as multi-stage retrieval, for the same reason: put the cheap filter where the volume is, and spend the expensive model only where the decision is genuinely in doubt.
Stage two is the interesting one. It runs on everything, so it sets your latency floor, and its job is not to decide — it is to route. Two thresholds, not one: below the low threshold allow, above the high threshold block, and in between escalate.
Why two thresholds beats one
A single threshold forces every borderline case into a wrong answer. Two thresholds let you set the confident bands aggressively — because you only apply them where the model is confident — and pay for real judgement on the small slice in the middle.
The width of that band is your cost dial. Widen it and you catch more, spend more, and slow more requests. Narrow it and the middle cases fall to whichever side you set. That dial is worth naming out loud; it is where the interviewer will push.
What each stage is good at
| Stage | Catches | Misses |
|---|---|---|
| Deny-list, regex | Known payloads, exact strings, structured PII | Anything paraphrased |
| Encoder classifier | Learned harm categories, fast, cheap | Nuance, context, novel framings |
| LLM judge | Context, intent, policy nuance | Nothing cheaply — it is the expensive tier |
The first stage catches almost nothing on adversarial traffic and is still worth having, because it is free and it removes the noise floor. The second does the real work. The third is where policy nuance lives — "is this a medical question or medical advice?" is not a keyword problem.
Parallel, not serial, where you can
Some checks do not gate each other. PII detection, language identification and topic classification can run concurrently with the harm classifier, and then the results combine. The stage-two latency is then the slowest check, not the sum.
Reserve serialisation for genuine dependencies: escalate to the judge only after the classifier is unsure, because that is the whole point.
Fail open or fail closed?
The guard is a dependency, and dependencies fail. When the classifier times out, do you allow the request or block it?
There is no universal answer, and saying so is the point. A consumer creative tool that fails closed becomes unusable during an incident, converting a guard outage into a product outage. A system that can dispense medical or financial guidance and fails open publishes unreviewed output for the duration.
The usable answer is per-surface and stated as policy: fail closed on the high-harm surfaces, fail open with degraded capability elsewhere — drop tool access, shorten max output, log everything for retrospective review. What you must not do is leave it undecided, because the default is whatever the HTTP client does on timeout.
Caching the decision
Guard verdicts cache well, and it is the cheapest win available. Identical inputs recur constantly, especially under attack — an attacker probing variations sends the same payload many times.
Key on a hash of the normalised input plus the policy version. The policy version matters: without it, a policy change silently keeps serving decisions made under the old rules, which is the same class of bug as an unversioned feature definition.
Cache blocks and allows both, with a short TTL on allows. And normalise before hashing — case, whitespace, homoglyphs — or the cache misses on exactly the traffic it was meant to absorb.
Key takeaway
Order guards cheapest-first and let a calibrated classifier route rather than decide: two thresholds, with the expensive judge seeing only the uncertain band. That keeps the median guard cost near ten milliseconds while retaining nuanced judgement where it is needed — and the width of the uncertain band is the single dial that trades cost against catch rate.
Next: why none of this solves prompt injection.