Throughput: Prefill, Decode, and Batching

In one line: a generative request is two workloads wearing one API, and almost every serving optimisation exists because they have opposite bottlenecks.

Two phases

Prefill processes the whole prompt. Every token can be handled in parallel because they are all already known, so the GPU does a large matrix multiplication and is compute-bound. It produces one token and the KV cache.

Decode generates the rest, one token at a time. Each step must read the entire model weights out of memory to produce a single token, so arithmetic intensity is terrible and the phase is memory-bandwidth-bound.

That split explains the two latency numbers users actually experience, and they are governed by different resources:

Which one matters depends on the product. A chat interface lives or dies on time-to-first-token, because that is the perceived responsiveness. A batch summarisation job cares only about total tokens per second. Saying which one you are optimising, and why, is the point of this lesson.

Batching helps decode enormously and prefill barely

This follows directly from the bottlenecks, and it is the fact that makes serving economics work.

During decode, the GPU reads all the model weights to produce one token for one request. If sixteen requests decode together, it reads those same weights once and produces sixteen tokens. The expensive part was the memory read, and it was shared — so throughput rises almost linearly with batch size for very little extra latency.

During prefill the GPU was already saturated with a large parallel matrix multiplication. Adding more work does not find idle capacity because there was none, so batching buys much less.

Continuous batching

The naive approach batches at request granularity: collect requests, run them together, wait for all to finish, start the next batch.

The problem is that generation lengths vary enormously. One request finishes in 20 tokens, another runs to 800. With static batching, the whole batch is held hostage by the longest member, and the slots belonging to finished requests sit idle producing nothing.

Continuous batching — also called iteration-level scheduling — makes the scheduling decision at every decode step rather than every request. A request that finishes frees its slot immediately, and a queued request joins on the next iteration.

Combined with paged cache allocation from the previous lesson, this is where the large reported throughput gains in modern serving stacks come from. The two are complementary: paging makes memory available, continuous batching keeps it occupied.

The interference problem, and disaggregation

Now put the two phases in one server and notice the conflict.

Prefill is a long compute-heavy burst. Decode is a stream of small memory-bound steps. When a new request's prefill lands on a server that is mid-decode for others, it stalls every in-flight stream — everyone's text visibly pauses because one person submitted a long prompt.

The mitigations, in increasing order of ambition:

Chunked prefill splits a long prompt into pieces and interleaves them between decode steps, so a large prompt degrades everyone slightly rather than stalling them completely.

Prefill/decode disaggregation goes further and assigns the phases to separate pools of hardware. Each pool is then sized and tuned for its own bottleneck, and a long prompt cannot interfere with anyone's decode.

The cost row is the honest one and the thing to name if you propose it: after prefill, that request's KV cache lives on the wrong machine and has to be transferred before decode can start. That transfer is real bandwidth and real latency, which is why disaggregation pays off at scale and is over-engineering below it.

Estimating throughput

A practical sequence for a serving estimate:

1. Decode throughput per GPU, batched   ~2,000 tokens/sec  (model-dependent)
2. Average completion length                   250 tokens
3. Completions per GPU per second       2,000 / 250  =  8
4. Peak completions per second needed                  400
5. GPUs for decode                      400 / 8      =  50
6. Add prefill capacity and headroom    ~ +30%
7. Total                                              ~65 GPUs

Then cross-check against the memory calculation from the previous lesson and take the larger of the two. Serving capacity is bounded by whichever binds first — memory for concurrency, or bandwidth for throughput — and which one it is depends on the traffic shape. Long conversations at low rate are memory-bound; short requests at high rate are throughput-bound.

Next: sizing embeddings and the vector index.