Failure, Degradation, and Observability
In one line: the question is not whether inference capacity will be unavailable but what the product does during those minutes, and the answer should be a ladder rather than an error page.
The degradation ladder
Design the rungs explicitly, from full quality down to something that still works.
Rung 5 is the one designs skip and the one that saves products. A support assistant that cannot generate can still **show the retrieved documents — worse than a written answer and enormously better than an error. A recommendation surface can fall back to popularity.
Two properties matter for every rung: it should be automatic, triggered by a signal rather than a human decision at 3am, and exercised regularly, because a fallback that has not run in six months is a hypothesis rather than a fallback.
Retries are more dangerous here
The standard retry-amplification problem, with an extra multiplier.
An ordinary failed request costs a little and is retried cheaply. A generation that fails after producing 800 tokens has already consumed real GPU work, and retrying it spends that again — so retries during overload consume capacity at several times the rate of the original requests.
| Ordinary service | LLM inference | |
|---|---|---|
| Cost of a failed attempt | Small — usually fails fast | Potentially a full generation already paid for |
| Can you scale out of a retry storm? | Usually | No — capacity is fixed |
| Effect of layered retries | Multiplies load | Multiplies load that cannot be absorbed |
| Right default | Retry with backoff | Retry only fast failures, with a budget |
The rule that follows: retry rejections and connection failures; do not blindly retry timeouts.** A timeout usually means the work was done or is still being done, so retrying it doubles the spend on a request that may yet succeed.
And use a retry budget — a cap on the fraction of traffic that may be retries — rather than a per-request count. A per-request limit still scales with traffic, so wider failure produces more retries exactly when there is least capacity. A budget is absolute, so wider failure produces proportionally fewer.
Timeouts need to match the work
A single timeout is wrong for a system where a request can legitimately take thirty seconds.
**Time to first token gets a tight timeout, because a long wait here means queueing or a stuck worker, and failing fast frees capacity.
Inter-token gets its own timeout, which catches a stalled stream that a total-duration timeout would not notice for another twenty seconds.
Total duration** is a backstop, generous, and should align with the client's own deadline — the queueing lesson's point that admitting work which cannot finish in time is worse than rejecting it.
What to log, per request
The observability requirement is specific, because the usual request log is not enough to explain anything here.
model and version, prompt version, index version prompt tokens, completion tokens, cached prefix tokens time to first token, total duration, queue wait which worker, cache hit or miss and which tier whether it was routed, downgraded, or escalated truncation flags, finish reason estimated cost
Three of those deserve a note.
**Queue wait separately from generation. Without the split, a latency regression is unattributable — you cannot tell an overloaded fleet from a slower model.
Finish reason. Whether a response ended because the model finished, hit the token cap, was cancelled or errored. A rising rate of length-capped completions is a quality problem hiding as a normal metric.
Cost per request as a first-class metric.** It drifts upward on its own — prompts accrete instructions, contexts grow, conversations lengthen — and teams that discover it from the invoice discover it a month late and cannot attribute it to a change.
The dashboards that matter
The third group is the one specific to this system and the one most often missing. Escalation rate decides whether routing is still saving money; semantic cache correctness decides whether it is still safe; cost per request catches both drifting before finance does.
Key takeaway
Build a degradation ladder — trim, downgrade, overflow to a hosted API, fall back to something non-generative, then fail honestly — with every rung automatic and regularly exercised. Retries are more dangerous here because a failed generation may already have consumed its full cost and capacity cannot grow, so retry fast failures only and bound them with an absolute budget rather than a per-request count. Set three timeouts, including the inter-token one everybody omits. And log queue wait separately from generation, the finish reason, and cost per request, because all three drift silently.
Next: the walkthrough, where the whole platform gets designed live.