Load Balancing Across GPU Workers

In one line: standard load balancing assumes workers are interchangeable, and an LLM worker holding a warm KV cache for your conversation is not.

What round-robin destroys

Consider a multi-turn conversation across four workers. Round-robin sends turn 1 to worker A, turn 2 to worker B, turn 3 to worker C.

Worker A already holds the KV cache for that conversation — the processed state of the entire prompt prefix. Worker B has none of it, so it must prefill the whole conversation from scratch, redoing work that already exists one machine away.

This compounds with the quadratic from the estimation chapter. Turn 10 carries nine turns of history, so a cache miss there is not a small penalty — it is prefilling thousands of tokens another worker had already processed.

The result is a fleet that looks correctly balanced on every dashboard while doing several times more prefill work than it needs to.

Prefix-aware routing

The fix is to route toward the worker that already holds the relevant cache. Requests sharing a prefix — the same conversation, or the same system prompt — go to the same worker so the prefill can be skipped.

The mature form is precise rather than heuristic: workers emit cache events, and the router maintains a trie of which prefixes live where, matching an incoming request's tokenised prompt against it to find the best overlap.

Two points in that diagram deserve emphasis, because they are where naive versions fail.

The tension: affinity versus load

Pure affinity is a trap. Send every request for a popular prefix to the one worker holding it and that worker saturates while the others idle — cache locality has become a hot spot.

So the routing decision is a **score combining both: how much prefix this worker already has, and how loaded it currently is. Routers considering both outperform pure session affinity with consistent hashing, precisely because they can decline a cache hit when taking it would overload the worker.

The second row is worth saying out loud: connection count is not load in this system. One connection generating 2,000 tokens over a 100K-token context is vastly more expensive than fifty short completions, so a balancer counting connections is measuring something nearly unrelated to cost.

The better load signal is tokens in flight or KV cache occupancy** — the things that actually constrain the worker.

Fall back rather than queue behind a cache hit

The rule that keeps the whole scheme safe.

When the ideal worker is saturated, routing to it anyway means the request waits — and the wait is usually longer than the prefill it would have saved. So the router must be willing to give up the cache hit and send the request to a free worker.

That single rule prevents cache locality from becoming a queueing disaster, and it is the difference between prefix-aware routing helping and hurting.

Three routing decisions, not one

Candidates often collapse these into "the load balancer picks a server". Separating them is a good structural signal.

Model selection is a cost decision and gets its own lesson. Pool selection separates workloads with different memory profiles — the estimation chapter's point that long-context requests deserve their own pool rather than sizing the whole fleet for the worst case. Worker selection is this lesson.

Next: not doing the work at all.