The Estimation That Matters: GPUs and Tokens

Why this matters: the previous lesson established that every resource the standard template measures is negligible here. This lesson computes the one that is not. Nothing else in the chapter constrains the design as tightly as what follows.

Step one: convert requests into tokens

The estimation cannot proceed in requests, because a request is not a unit of work here. The unit is a token — roughly three quarters of an English word, or about four characters.

Response size          = 5 KB
Tokens per response    = 5,000 chars / 4 chars per token  = 1,250 tokens
Aggregate token rate   = 17,361 req/s x 1,250 tokens      = 21.7 M tokens/second

A 5 KB response is a very long answer — and that reframes the latency requirement

1,250 tokens is roughly 900 words, several screens of text. The source picked 5 KB casually as "AI-generated replies are typically longer," but it implies a substantial essay on every one of 1.5 billion daily requests.

That matters because of what it does to latency. At a comfortable reading pace of 50 tokens per second:

1,250 tokens / 50 tokens per second = 25 seconds

Twenty-five seconds to finish generating. At 20 tokens per second it is over a minute.

Compare that with every latency budget earlier in the course — Google Maps wanted routes in 2 to 3 seconds, and that was considered demanding. Here the response is not even complete for half a minute.

This is what makes streaming non-negotiable rather than a nicety. Nobody waits 25 seconds for a blank screen. Tokens are sent as they are produced, so the user starts reading at roughly one second while generation continues underneath.

And it is why the meaningful latency metrics are a pair rather than a single number:

Time to first token (TTFT) — how long until something appears. This is the number users experience as responsiveness. Target: under a second.
Time per output token (TPOT) — the pace afterwards. It only needs to beat reading speed, so roughly 30 to 50 tokens per second is sufficient.

Optimizing them pulls in opposite directions, which is the central tension of the rest of this lesson.

Step two: prefill and decode are different workloads

Generating a response has two phases with completely different performance characteristics.

	Prefill	Decode
What it does	Processes the entire prompt	Generates output tokens one at a time
Parallelism	All prompt tokens at once	Strictly sequential — token N needs token N-1
Forward passes	One	One per output token — 1,250 of them
Bottleneck	Compute (FLOPs)	Memory bandwidth
Determines	Time to first token	Time per output token
GPU utilization	High — the hardware is busy	Low — the hardware waits on memory

Why decoding is memory-bound — the single most important fact in LLM serving

This is unintuitive and it explains nearly every serving optimization, so it is worth deriving.

To generate one token, the model must perform a forward pass. A forward pass touches every parameter in the model. Those parameters live in GPU memory and must be read into the compute units.

Weights to read per token = 70B params x 2 bytes = 140 GB
Memory bandwidth          = 3,350 GB/s
Maximum token rate        = 3,350 / 140 = 23.9 tokens/second

Twenty-four tokens per second is the ceiling for a single sequence — not because the arithmetic is slow, but because you cannot pull 140 GB from memory any faster.

And here is the waste: with one user, that entire 140 GB read produces exactly one token. The GPU's arithmetic units are almost entirely idle, waiting on memory.

Now the crucial observation. If you process many sequences simultaneously, you read those 140 GB of weights once and use them for every sequence in the batch:

Batch size	Weight reads	Aggregate throughput	Per-user rate
1	140 GB per token	24 tok/s	24 tok/s
8	140 GB per 8 tokens	191 tok/s	24 tok/s
32	140 GB per 32 tokens	766 tok/s	24 tok/s
128	140 GB per 128 tokens	3,063 tok/s	24 tok/s
256	140 GB per 256 tokens	6,126 tok/s	24 tok/s

Throughput scales nearly linearly with batch size while per-user speed stays constant. That is a remarkable property and it has no analogue in conventional serving, where adding concurrent requests to a saturated server makes everyone slower.

The reason it works is that decode is bandwidth-bound and the batch is not — the weights were going to be read regardless, so extra sequences ride along for free until the arithmetic units themselves saturate.

This is why batching is not an optimization here. It is the difference between a viable service and an impossible one.

Step three: the KV cache, and why it bounds everything

If large batches are free throughput, why not batch a thousand sequences? Because each one occupies memory that grows with the conversation.

The KV cache is the reason context length is expensive

Recall from Lesson 1 that generating token N requires attending to tokens 1 through N-1. Recomputing that from scratch each step would be quadratic and hopeless, so the intermediate key and value tensors for every previous token are kept in GPU memory. That store is the KV cache.

For a 70B-class model with 80 layers and a model dimension of 8,192:

KV per token = 2 (keys and values) x 80 layers x 8,192 x 2 bytes = 2.62 MB

Per token. Per sequence. So:

Context length	KV cache per sequence
1,000 tokens	2.6 GB
4,000 tokens	10.5 GB
32,000 tokens	84 GB
128,000 tokens	336 GB

A single sequence at 128k context needs more memory than four GPUs, purely for its cache.

Modern models reduce this substantially with grouped-query attention, which shares key and value tensors across groups of attention heads. At 8 KV heads for 64 query heads the cache drops eightfold — 4,000 tokens of context falls from 10.5 GB to about 1.3 GB.

Now do the batching arithmetic honestly. Two 80 GB GPUs give 160 GB. The weights take 140 GB. That leaves 20 GB for KV cache:

20 GB / 1.3 GB per sequence = about 15 concurrent sequences

Fifteen. Not the 256 the throughput table wanted.

This is the real constraint, and it is why serving stacks are memory-allocation systems as much as inference systems. The fixes are all about buying back KV space: add GPUs so weights are a smaller fraction of total memory, quantize the weights, quantize the cache itself, page it in fixed blocks so fragmentation does not waste space, or simply serve shorter contexts.

It also gives a hard-nosed answer to a product question. "Why does a longer context window cost more?" Not because of a pricing decision — because KV cache scales linearly with context length and directly displaces the concurrency that pays for the hardware.

Step four: size the fleet

Aggregate demand    = 21.7 M tokens/second
Per-GPU throughput  = ~2,000 tokens/second (batched, mid-size model)
GPUs required       = 21.7M / 2,000 = about 10,900 GPUs

Per-GPU throughput	GPUs needed
1,000 tok/s	21,700
2,000 tok/s	10,900
4,000 tok/s	5,400

Put that beside 0.27 web servers

The previous lesson's corrected estimate said the web tier needs less than one server. The inference tier needs roughly ten thousand GPUs — and at data-centre prices that is capital in the billions before electricity.

Two things follow, and both are worth saying out loud in an interview.

First, the ratio between the two numbers is the design. Every architectural decision in this chapter should be evaluated by whether it moves work from the ten-thousand-GPU side to the less-than-one-server side. Semantic caching does. Routing simple prompts to a small model does. Adding a feature that requires an extra model call does the opposite, and needs to justify itself against roughly a dollar-per-thousand-requests of hardware.

Second, doubling per-GPU throughput halves the fleet. Look at the table: the difference between 1,000 and 4,000 tokens per second per GPU is 16,000 GPUs. That is why the batching and KV-cache mechanics above are not implementation trivia — a scheduling change that raises average batch size is worth more than any amount of application-tier optimization.

This is the inverted priority that defines AI systems. In every previous chapter, the expensive tier was storage or bandwidth and compute was noise. Here compute is four orders of magnitude more expensive than everything else combined, so effort belongs there and nowhere else.

Step five: the scheduling problem

Fixed batching wastes enormously here, for a reason specific to generation: sequences in a batch finish at different times. One user asks for a word, another for an essay. With static batching the whole batch waits for the longest.

Continuous batching — evict and admit at every step

The fix is to treat the batch as a living set rather than a fixed group. At every decoding step, completed sequences are evicted and waiting requests are admitted into the freed slots.

The scheduler is deciding, thousands of times a second, how many sequences to run given remaining KV memory — and it faces the tension from the start of this lesson directly:

Prefill is compute-heavy and produces the first token. Prioritizing it improves TTFT for new arrivals.
Decode is bandwidth-bound and continues existing responses. Prioritizing it improves TPOT for users already reading.

They contend for the same hardware. Admit too many new requests and existing users watch their text stutter; admit too few and new users stare at a blank screen. Production schedulers mix the two deliberately, and this trade-off is worth naming — it is the clearest example in the chapter of a genuine engineering choice with no free answer.

How this changes the standard scaling answers

Several reflexes from earlier chapters behave differently here, and knowing which ones is a good marker of having thought about it.

Autoscaling is slow. Adding a web server takes seconds. Adding an inference server means loading 140 GB of weights into GPU memory — minutes, not seconds. So capacity must be provisioned ahead of demand, and the smooth reactive scaling assumed elsewhere does not apply.

Load balancing is not round-robin. Requests are wildly unequal — a 50-token reply and a 2,000-token reply cost forty times as much and neither is knowable in advance. Sensible routing tracks KV memory occupancy and active sequence count per server, not request counts.

Queuing is normal, not a failure. Because GPU capacity is fixed and expensive, running at high utilization is the point. A short queue is a sign of correct provisioning. What matters is that queue time is counted inside TTFT, since the user cannot tell waiting from thinking.

The failure mode is memory, not CPU. A server does not slow down gradually as it fills — it runs at full speed until KV cache is exhausted, then must either refuse admissions or evict a sequence mid-generation. Admission control is therefore load-bearing rather than defensive.

Key takeaway

The unit of work is a token, and 17,361 requests per second is 21.7 million tokens per second. Generation splits into prefill — one parallel, compute-bound pass over the prompt, setting TTFT — and decode, 1,250 sequential, memory-bandwidth-bound passes setting TPOT. Because decode must read all 140 GB of weights per token, a single sequence is capped near 24 tokens/second, and batching multiplies throughput at no cost to per-user speed — which makes it the difference between viable and impossible. The limit on batch size is the KV cache, which grows linearly with context and can leave room for only a dozen sequences. The fleet is roughly 10,900 GPUs against 0.27 web servers, and that ratio is the design: every good decision moves work from the expensive side to the cheap one.

Interview signal by level

Level	What a strong answer sounds like
L4	"We need GPU servers for inference and we'd scale them horizontally behind a load balancer."
L5	Converts to tokens and knows the phases: "the unit is tokens, not requests — 21.7M tokens per second. Prefill processes the prompt in one pass and sets time-to-first-token; decode is sequential and sets the pace after. We batch to keep GPUs busy and stream tokens so users see output immediately."
Staff+	Derives the bottleneck: "decode is memory-bandwidth bound — every token requires reading all 140 GB of weights, capping a single sequence near 24 tokens per second. Batching amortizes that read, so throughput scales with batch size at constant per-user speed. The limit is KV cache: about 1.3 GB per sequence at 4k context with grouped-query attention, and if weights leave only 20 GB free you fit fifteen sequences, not 256. So the serving stack is really a memory allocator — paged KV cache, quantization, continuous batching that evicts and admits every step. And the scheduler trades prefill against decode, which is TTFT against TPOT. Fleet is ~10,900 GPUs versus 0.27 web servers, so I'd evaluate every feature by whether it moves work off the GPU side."

Next: the building blocks, and the high-level flow.