Multi-Tenancy, Quotas, and Fairness
In one line: requests per second is not a unit of consumption here, so a rate limiter built on it will let one tenant take the whole fleet while appearing to be within limits.
Why request-based limits fail
A tenant limited to 100 requests per second can send 100 requests each with a 100,000-token context and a 4,000-token completion. Another sends 100 requests of 50 tokens each. Both are "at their limit"; one is consuming three orders of magnitude more capacity.
The unit has to be the thing that is actually scarce.
Limit on tokens, and on both kinds
The primary budget is **tokens per unit time, and it should distinguish prompt from completion because they consume different resources — prefill is compute, decode is bandwidth and cache occupancy.
| Limit | Protects against | Missing it means |
|---|---|---|
| Tokens per minute | Sustained overconsumption | One tenant can absorb the fleet |
| Concurrent requests | KV cache exhaustion | A tenant holds memory others need |
| Max context per request | A single request taking a whole GPU | One 128K request evicts many small ones |
| Max output length | Unbounded generation cost | A runaway loop bills indefinitely |
| Requests per second | Gateway and cache abuse | Cheap-request floods, but not capacity abuse |
The third row is the one specific to this system. Because concurrency is memory-bound, a single very long-context request consumes what dozens of ordinary ones would — so a per-request context cap is a fairness mechanism, not just a product limit.
The last row is worth keeping despite everything above: request-rate limits still have a job, they are just protecting a different resource.
The accounting problem
Token limits have an awkward property: you do not know the cost until the request is finished.** Prompt tokens are countable up front; completion tokens are not.
The standard resolution is a reservation:
Reserving the maximum and settling the difference is conservative and correct. It briefly over-charges a tenant who asked for a large max_tokens and generated little, which is the right direction to be wrong — the alternative admits work you cannot pay for.
The noisy neighbour is structural
On elastic infrastructure, one heavy tenant mostly costs money. On a fixed GPU fleet, one heavy tenant **takes capacity from everyone else, because there is no additional capacity to take.
Three isolation levels, and the choice is a business decision as much as a technical one:
Shared pool with quotas. Cheapest and highest utilisation. Quotas bound the damage, and a tenant within quota can still degrade others during a spike, because quota is about sustained rate rather than instantaneous contention.
Reserved capacity per tenant. Dedicated workers for large customers. True isolation, and it strands capacity when that tenant is idle — which is most of the time.
Hybrid.** Reserved floor plus shared burst. A tenant is guaranteed their floor and can use spare shared capacity above it. Usually the right answer, and worth proposing as the design rather than as a compromise.
Fairness within a tenant
The subtler failure, and one candidates rarely reach.
A tenant within their quota can still starve their own users if one workload dominates. A batch job inside a customer's account consuming their entire token budget means their interactive users see nothing — and from the platform's perspective everything is compliant.
The fix is hierarchical: quota at the tenant level, and priority classes within it. The tenant's interactive traffic is protected from the tenant's own batch traffic, using the same class mechanism from the queueing lesson applied one level down.
What a rejection should say
Rejections are a product surface, and a good one prevents the retry storm that turns a limit into an outage.
Say which limit was hit, so the client can act on the right one. Give a retry-after hint, so clients back off in a coordinated way rather than all returning at once. Distinguish quota exhaustion from capacity pressure — the first means wait for the window, the second means try again shortly — because the correct client behaviour differs.
Key takeaway
Requests per second is not a unit of consumption when requests differ in cost a thousandfold, so budget on tokens — split by prompt and completion — plus concurrency, per-request context and output caps. Because completion cost is unknown up front, reserve an estimate at admission and settle on completion. On fixed capacity the noisy neighbour takes capacity rather than money, so give large tenants a reserved floor and let everyone burst into a shared pool. Apply priority classes inside a tenant too, and make rejections say which limit was hit and when to return.
Next: what happens when all of it fails.