KV Cache: Why Concurrency Is Memory-Bound

In one line: every in-flight request holds a block of GPU memory that grows with its conversation length, so your concurrency limit is leftover memory divided by per-request cache — and that is the number that surprises people.

Why the cache exists

Generation is autoregressive: the model produces one token, appends it, and runs again. Without a cache, generating token 500 would mean recomputing attention over all 499 previous tokens from scratch, and the total work would grow with the square of the output length.

The KV cache stores the key and value tensors for every token already processed, so each new token attends against stored state rather than recomputing it. It converts a quadratic amount of compute into a linear amount of memory.

That trade is the whole story: the cache buys speed with memory, and memory is the resource you run out of.

The formula

KV bytes per token  =  2  x  layers  x  heads  x  head_dim  x  bytes
                       ^
                       one for K, one for V

Worked for a 70B-class model using grouped-query attention — 80 layers, 8 key-value heads, head dimension 128, FP16:

2 x 80 x 8 x 128 x 2 bytes  =  327,680 bytes  ~=  0.31 MB per token

Then scale it by context length:

The last row is the one to sit with. A single 128K-context request consumes half an 80GB card, on top of weights that already needed two cards. One user, one conversation, half a GPU.

The concurrency consequence

Now combine it with the leftover memory from the previous lesson.

Work an example on a single card with a 7B model:

80 GB total
-14 GB weights
- 5 GB runtime overhead
=61 GB for cache

At 4K context, ~0.05 MB/token for a 7B model:
  4,000 x 0.05 MB  =  ~0.2 GB per request
  61 / 0.2         =  ~300 concurrent requests

At 128K context:
  128,000 x 0.05 MB = ~6.4 GB per request
  61 / 6.4          = ~9 concurrent requests

Same hardware, same model, same everything — 300 concurrent requests versus 9, decided entirely by context length. A thirty-fold capacity swing from a product decision that looks like a feature flag.

Why grouped-query attention exists

Look again at the formula and notice which term is doing the damage: the number of key-value heads.

Classic multi-head attention gives every query head its own key and value heads. Grouped-query attention lets several query heads share one key-value head, which divides the cache by the sharing ratio.

The point worth making in an interview: GQA is a serving optimisation baked into the model architecture. It exists because someone did exactly the capacity calculation above and concluded that the cache, not the weights, was the ceiling. Recognising an architectural choice as a response to an infrastructure constraint is a good signal.

What PagedAttention fixes

The naive implementation reserves a contiguous block per request, sized for the maximum possible length. A request that could run to 4,000 tokens reserves 4,000 tokens of cache the moment it starts, even if it finishes in 200.

The result is that most reserved memory is never used — fragmentation and over-reservation waste a large fraction of the cache, commonly cited as 60–80%.

The idea is virtual memory applied to attention: allocate small fixed-size blocks on demand and map them through a block table, so physical memory need not be contiguous. Recovering most of that wasted 60–80% translates almost directly into concurrency, which is why it produced a step change in serving throughput.

The second benefit is prefix sharing. If a thousand requests begin with the same system prompt, the blocks holding it can be shared rather than duplicated — which is the mechanism underneath prompt caching, and another reason to put the stable part of a prompt first.

Estimating a fleet

Putting the chapter together:

Peak concurrent conversations      10,000
Average context                     8,000 tokens
Cache per request (7B, GQA)        ~0.4 GB
Cache needed                       10,000 x 0.4  =  4,000 GB
Usable cache per card              ~61 GB
Cards needed                       4,000 / 61    =  ~66 GPUs

Then state the sensitivity, because that is what makes it a design conversation rather than a number: at 4K average context this halves to about 33 GPUs; at 32K it quadruples to roughly 260. Context length is the single biggest lever on fleet size, and it is usually decided by someone who has never seen this calculation.

Next: throughput, and why prefill and decode are different problems.