The Queue Is the Design
In one line: you cannot add GPUs during a spike, so the only question is which requests get served and which are told no — and a queue that answers that badly makes overload worse rather than better.
Why queueing is more dangerous here
The instinct from ordinary systems is that a queue absorbs bursts. That holds when the burst is short and capacity can grow. Here neither is reliably true.
A request sitting in a queue for eight seconds, then generating for six, has taken fourteen seconds. If the client gave up at ten, the system spent six seconds of scarce GPU capacity producing an answer nobody receives — and it did so while other requests were waiting.
That is the mechanism behind the classic failure: under sustained overload, a deep queue means the system does progressively more work that is already worthless, so effective throughput falls exactly when it is needed most. The backlog then grows faster, and the system does not recover on its own.
The client timeout sets the queue depth
The fix follows directly. If a request cannot start early enough to finish before the client gives up, it should never start.
Max useful queue wait = client timeout - expected generation time - margin
Client timeout 30 s
Expected generation 10 s
Margin 3 s
------
Max queue wait 17 s
Then bound the queue by that:
Max queue depth = max useful queue wait x throughput (requests/sec)
That is a derivation rather than a guess, and it is the kind of thing worth doing out loud. A queue sized this way cannot fill with requests that are already doomed.
Also check deadlines at dequeue, not only at enqueue. A request that was viable when it arrived may not be by the time a slot opens — checking again before admitting it costs nothing and reclaims real capacity.
Priority classes
One queue treats a paying customer's interactive request and a background summarisation job identically. Separating them is the highest-value structural change available.
| Class | Latency need | Under pressure |
|---|---|---|
| Interactive, paid | Strict — a human is waiting | Protect. Shed last |
| Interactive, free tier | Moderate | Degrade to a cheaper model first |
| Background and batch | None | Pause entirely; it is what fills the trough |
| Internal, evaluation | None | Pause, and run off-peak by default |
The third row is the one that pays twice. Batch work exists to fill idle capacity, so pausing it during a spike costs nothing that matters and returns capacity immediately. A platform without that lever has to take the spike out of interactive traffic instead.
Shedding beats slowing
When capacity is genuinely exceeded, the choice is between serving fewer requests properly and serving all of them badly. The second is worse, and it is the default if you do nothing.
Shedding well means being explicit about it: return a status the client can act on, include a retry hint so clients back off in a coordinated way, and shed by class rather than at random. A clear rejection is a better product experience than a request that hangs for thirty seconds and then fails anyway.
Degrade before you shed
There is a rung between "serve normally" and "reject", and it is usually the right first move.
Route to a smaller model. The answer is worse and it exists.
**Reduce the work per request. Retrieve four chunks instead of eight; cap the output length; skip the reranker.
Disable the expensive optional path.** Turn off the second-pass verification, or the follow-up suggestion generation.
Each frees capacity without refusing anyone, and a degradation ladder that runs from full quality down to a static fallback is a stronger answer than a binary serve-or-shed.
Admission control as one decision
Pulling it together, the gateway makes a single decision per request with everything it knows:
The ordering matters: check quota first because it is cheapest, then feasibility, then degradation. Every rejection that happens before a GPU is touched is capacity returned to requests that can use it.
Key takeaway
Fixed capacity makes overload a certainty, and a deep queue makes it worse — under sustained load the system spends increasing GPU time on requests whose clients have already given up, so effective throughput falls exactly when it is needed. Derive queue depth from the client timeout minus expected generation time, and re-check the deadline at dequeue rather than only at arrival. Separate priority classes so batch work can be paused to return capacity instantly, reserve a floor against starvation, and degrade — smaller model, less retrieval, shorter output — before shedding, then shed explicitly with a retry hint rather than letting requests hang.
Next: which worker the admitted request should go to.