Constrained Decoding and Structured Output
In one line: instead of asking a model to produce valid JSON and hoping, you can make invalid JSON unrepresentable — and that turns a probabilistic contract into a structural one.
How it works
At each decoding step the model produces a distribution over the whole vocabulary. Constrained decoding compiles your schema into a state machine, and at every step that machine determines which tokens could still lead to a valid output.
Everything else has its logit set to negative infinity, so it cannot be sampled.
The consequence is categorical rather than statistical. You never get a parse error, never retry for format, and never need a fallback parser. A whole class of production failure disappears.
What that replaces
Without it, the usual arrangement is a chain of hopes and patches:
| Prompt-and-hope | Constrained decoding | |
|---|---|---|
| Validity | Probable | Guaranteed |
| Failure handling | Retry, repair, or a lenient parser | Not needed |
| Cost of a failure | A full extra generation | None |
| Tail behaviour | Rare malformed outputs reach production | Structurally impossible |
| Overhead | None at generation time | A few percent per step |
The fourth row is the one that matters most. Prompting gets format compliance to something like 97%, which sounds fine and means thirty thousand malformed responses per million — arriving unpredictably, in the tail, usually on the hardest inputs.
The overhead is small
Masking at every step costs something: typically a few percent for regex or JSON-schema constraints, more for deep nested grammars. In practice it is rarely the bottleneck, and modern implementations reduce it further by caching the context-independent parts of the mask computation.
Compared against the retry it eliminates — a full second generation at maybe 3% of requests — constrained decoding is usually **cheaper on average, not more expensive. That is a good line to have ready, because the instinct is to treat it as a tax.
The constraint tax is real, though
Here is the honest caveat, and volunteering it is what separates understanding the mechanism from repeating a vendor claim.
Forcing the output into a rigid structure can degrade the quality of the content inside it.** A model constrained to emit JSON immediately has no room to reason before answering, and research on format restrictions has found measurable performance drops on reasoning-heavy tasks under tight constraints.
The standard fix is to give the reasoning somewhere to live. Put a free-text field first in the schema — a reasoning property the model fills before the structured fields — so the constraint still holds and the model can think inside it. Or split into two calls: reason freely, then extract under constraint.
Field order in the schema matters for the same reason: fields are generated in order, so anything the model should decide after thinking must come after the thinking field.
Where it belongs
Not everywhere. The decision is simple:
**Use it whenever a machine consumes the output. Tool and function calling, structured extraction, classification into a fixed label set, routing decisions, anything feeding a parser.
Skip it when a human consumes the output.** Prose answers, summaries, explanations — there is no schema, and constraining fluent text buys nothing.
The first category is larger than it looks in an agentic system, where most model calls are decisions rather than prose. Tool calling is the clearest case: an invalid tool call is not a degraded answer, it is a crash, and the arguments must match a signature exactly.
Key takeaway
Constrained decoding compiles a schema into a state machine and masks every token that would break validity, so malformed output becomes impossible rather than improbable — eliminating parse errors, retries and fallback parsers for a few percent per-step overhead that is usually cheaper than the retries it removes. The real cost is the constraint tax: a rigid structure from the first token leaves no room to reason, so put a free-text field first and remember that fields generate in order. And guaranteed-valid structure says nothing about whether the contents are true.
Next: scaling a fleet that cannot scale in seconds.