The Threat Model

In one line: you cannot design a guard layer until you name what it is guarding against, and the six failures differ enough that no single mechanism covers two of them well.

Start by asking who the adversary is

There are three, and conflating them produces a design that serves none.

The curious user is not attacking you. They ask something the policy forbids, get refused, and move on. Volume is high, intent is low, and a cheap classifier handles almost all of it.

The determined attacker is iterating. They will try a hundred phrasings, encode the request, translate it, wrap it in a fictional frame. No filter survives unlimited attempts, so the design goal against this adversary is cost — make each attempt slow, logged and rate-limited, so persistence becomes expensive rather than free.

The third party never touches your interface. They write a web page, a document, an email, a support ticket — and wait for your system to read it. They are attacking through your retrieval, not your input box, and every input-side guard you built is on the wrong side of the door.

Say this split out loud in an interview. It shows you know that most of your traffic is not adversarial and that your hardest adversary never appears in your request logs as an attacker.

The six failures

Harmful content

The model produces something that violates policy — dangerous instructions, harassment, content involving minors, regulated advice. This is the failure everyone designs for, and it is the one that classifiers genuinely address.

Prompt injection

Text the model reads is treated as an instruction. Direct injection comes from the user; indirect injection comes from content the system retrieved on the user's behalf. The second is the serious one and gets its own lesson.

Data leakage

The model reveals something it should not: another tenant's data pulled in by a retrieval bug, a system prompt containing a key, PII echoed into a log, a customer record surfaced to the wrong support agent. Note that most leakage paths are not the model's fault — it faithfully repeated what it was given.

Excessive agency

The system takes an action nobody sanctioned. OWASP added this as its own category in the 2025 LLM Top 10, and split the root causes three ways: too much functionality, too many permissions, too much autonomy. A model with a send_email tool and a compromised context is a mail server for whoever wrote the document it read.

Improper output handling

The application executes what the model produced — rendering it as HTML, passing it to a shell, interpolating it into SQL. This is a classical injection vulnerability with a language model as the untrusted source, and it is why "never execute model output" is a rule rather than a preference.

Cost and availability

Long inputs, forced long outputs, or a loop of retries turn your inference budget into an attack surface. A single request can cost thousands of times what a typical one costs, which makes per-user token budgets a safety control rather than a billing detail.

Which of these can a filter address?

This table is the reason the threat model comes before the architecture.

One of six is a filtering problem. The rest are architecture: where the trust boundary sits, what identity a tool runs as, what the application does with a string it received.

The multi-turn hole

One more, because it is where single-turn designs break.

Every guard so far inspects one message. Real attacks are staged: establish an innocuous frame, build context, ask the question three turns later when it reads as continuous. Each turn passes on its own. The conversation does not.

The defence is to evaluate the conversation, not the turn — score the accumulated context, and re-score when the topic shifts. It costs more, and you cannot skip it in any product with memory.

What this buys you in the room

Scoping is scored. A candidate who opens with "what are we protecting against, and from whom?" and then narrows — say, a consumer assistant with retrieval and two write tools, so injection and agency dominate and harmful content is the smaller risk — has already demonstrated the judgment the rest of the answer is built on.

Next: the input guard cascade — how to run several checks inside one latency budget.