Content Moderation
In one line: It is worth answering properly, because the honest answer is that the premise is slightly wrong.
Why is content moderation placed after response generation rather than before?
The direct answer
The tension is real and worth naming: streaming and output moderation want opposite things. The usual resolution is to moderate in windows as tokens are produced, accepting that a bad phrase can appear briefly before being retracted.
Because the risky text is the text the model wrote
Moderation exists to stop harmful content reaching a user. The content that reaches a user is the model's output. So output is what must be checked — screening only the input would leave the actual delivered text unexamined.
Three reasons the input is not a sufficient proxy:
Benign prompts produce harmful outputs. "Write a realistic villain's monologue" is an ordinary creative request that can generate genuinely unpleasant text. Nothing in the prompt predicts it.
Harmful intent hides behind innocuous phrasing. Prompt injection, role-play framings, and incremental multi-turn escalation are all designed to look harmless at the input. A filter on the prompt would pass them.
Generation is not deterministic. Even if you could predict the output distribution for a prompt, sampling means this response is not necessarily the typical one. Only inspecting what was actually produced tells you what the user will actually see.
So output moderation is the check that corresponds to the actual risk. It is placed after generation because that is the only place the object it inspects exists.
Why the question's premise is incomplete
It should not be after instead of before — it should be both
The question presents an either/or. Production systems run input moderation and output moderation, and they do different jobs.
| Input moderation | Output moderation | |
|---|---|---|
| Inspects | The user's prompt | The generated response |
| Catches | Clearly disallowed requests, prompt injection, known attack patterns | Harmful, unsafe, or policy-violating text however it arose |
| Saves | The entire cost of generation | Nothing — the compute is already spent |
| Can it be sufficient alone? | No — benign prompts yield harmful output | Nearly, but it wastes GPU time on requests that should never have run |
The cost column is the argument the design misses entirely, and it is the strongest one available given everything Lesson 4 established.
If a prompt is obviously disallowed, generating 1,250 tokens before rejecting it burns seconds of GPU time from a ten-thousand-unit fleet to produce something thrown away. Input screening is a cheap classifier — milliseconds — that avoids the expensive path entirely.
This is the same principle as the semantic cache from Lesson 7, applied to a different case: anything that can avoid invoking the model pays for itself immediately. Input moderation is not primarily a safety optimization, it is a cost one, and the safety benefit comes along with it.
So the accurate answer to the design's question is: "moderation is placed after generation because output is what reaches the user and benign prompts can produce harmful output — but it should not be only after. Input screening runs first, primarily because rejecting before generating saves the most expensive resource in the system."
The problem streaming creates
You cannot moderate a complete response that has not finished existing
Here is the tension the design's placement quietly creates, and it has no clean resolution.
Lesson 9 established that responses stream — tokens go to the user as they are produced, because a 25-second wait is unacceptable. But moderation as described "scans generated responses before delivery," which requires a complete response.
Those two requirements are in direct conflict. Pick one and you lose the other:
Moderate the complete response, then deliver. Safe and simple. But it reintroduces the full 25-second wait, destroying the responsiveness streaming existed to provide. Time to first token becomes time to last token.
Stream first, moderate after. Fast, but the harmful text has already been read. Retracting it — some products do blank the response mid-stream — is visible, jarring, and arguably worse than never showing it.
Moderate incrementally, in windows. The practical compromise: buffer a small number of tokens, classify the window, release it, continue. Keeps latency near-streaming with a slight delay, and catches most problems early.
Its weakness is that harm is not always local. A passage can be innocuous sentence by sentence and unacceptable as a whole, and a window classifier cannot see the whole. So incremental moderation catches locally detectable problems and misses globally emergent ones.
The honest position is that all three options are unsatisfying, and real systems layer them — input screening to avoid the worst cases, incremental checks during streaming, and a full-response check whose result may arrive after the user has read it, used for logging, retraining, and account action rather than prevention.
Being able to say "there is no clean answer here and here is the layered compromise" is a much stronger interview signal than confidently describing one of the three as though the other two did not exist.
Moderation is a model too — with all that implies
Easy to picture as a keyword blocklist. It is not; it is a classifier, usually a small language model, which means it inherits every property of the thing it guards:
It has false positives and false negatives, and they trade against each other. Tighten it and legitimate medical, legal, or security questions get refused — the over-refusal failure users find infuriating. Loosen it and harmful content slips through.
It costs compute. Small compared to generation, but it runs on every response, so it is a real fraction of the fleet.
It can be attacked. Adversarial phrasing can evade a classifier the same way it can manipulate a generator.
It needs its own evaluation. Precision and recall on a labelled set, monitored over time, because both the model and the attacks drift.
The recursion is worth naming out loud: we are using a language model to check a language model, with no ground truth for either. It is the best available answer, and it is not a guarantee. A candidate who says that sounds more credible than one who treats moderation as a solved box in the diagram.
What moderation does not do
Worth stating plainly, because it is the gap most often glossed.
"* Not for truth.
A response can be perfectly compliant — no violence, no harassment, no disallowed material — and completely false. Moderation will pass it, because it is checking a different property.
That closes the loop with Lesson 2's observation. Safety appears in the architecture but not in the requirements. Accuracy appears in neither. The design has a component for harmful content and no component, anywhere, for wrong content.
That is not an oversight to fix with another box in the diagram. Verifying factual claims in open-domain generated text is an unsolved problem. The partial mitigations — grounding answers in retrieved sources and citing them, which the next lesson covers — reduce it without solving it.
Key takeaway
Moderation runs after generation because the output is what reaches the user and benign prompts can produce harmful text. But the premise is incomplete: it should run on both sides, and input screening's real argument is cost — rejecting before generating saves seconds of GPU time, the same "avoid invoking the model" principle as the semantic cache. Streaming makes complete-response moderation impossible without destroying the latency it exists to provide, so real systems layer input screening, incremental windowed checks, and a post-hoc full-response check used for logging rather than prevention. Moderation is itself a model, with false positives, compute cost, and no ground truth — and it checks for policy violations, not truth.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Moderation checks the response before we send it, so harmful content doesn't reach the user." |
| L5 | Explains why output must be checked: "a benign prompt can produce a harmful response and generation isn't deterministic, so screening the input alone doesn't tell you what the user will see." |
| Staff+ | Corrects the framing and owns the conflict: "it shouldn't be after instead of before — it should be both, and the argument for input screening is cost: rejecting a disallowed prompt before generating saves seconds of GPU time on a ten-thousand-GPU fleet. The harder issue is that moderation wants a complete response and streaming means there isn't one — so you either kill responsiveness, retract text the user already read, or moderate in windows and miss globally emergent harm. There's no clean answer; I'd layer all three. And moderation is a classifier with its own precision-recall trade, so tightening it produces over-refusal on legitimate medical and security questions. It also checks policy, not truth — nothing in this design detects a confidently wrong answer." |
Next: context and retrieval — how the system knows what the user is talking about.