The Estimation That Matters: GPUs and Tokens

In one line: the previous lesson established that every resource the standard template measures is negligible here. This lesson computes the one that is not. Nothing else in the chapter constrains the design as tightly as what follows.

A note on this lesson

Most treatments of this problem compute storage and bandwidth, note that generative AI leans heavily on GPU inference, and move on. This lesson exists to fill that gap, because a design for an LLM system that never sizes its inference tier has not been designed.

The figures use a 70-billion-parameter model in FP16 on an 80 GB accelerator with 3.35 TB/s of memory bandwidth — roughly a current data-centre GPU. Treat them as order-of-magnitude reasoning, not as specifications of any real deployment. The relationships matter more than the constants.

Step one: convert requests into tokens

The estimation cannot proceed in requests, because a request is not a unit of work here. The unit is a token — roughly three quarters of an English word, or about four characters.

Response size          = 5 KB
Tokens per response    = 5,000 chars / 4 chars per token  = 1,250 tokens
Aggregate token rate   = 17,361 req/s x 1,250 tokens      = 21.7 M tokens/second

A 5 KB response is a very long answer — and that reframes the latency requirement

1,250 tokens is roughly 900 words, several screens of text. The usual figure of 5 KB gets picked casually — "AI-generated replies are typically longer" — but it implies a substantial essay on every one of 1.5 billion daily requests.

That matters because of what it does to latency. At a comfortable reading pace of 50 tokens per second:

1,250 tokens / 50 tokens per second = 25 seconds

Twenty-five seconds to finish generating. At 20 tokens per second it is over a minute.

Compare that with every latency budget earlier in the course — Google Maps wanted routes in 2 to 3 seconds, and that was considered demanding. Here the response is not even complete for half a minute.

This is what makes streaming non-negotiable rather than a nicety. Nobody waits 25 seconds for a blank screen. Tokens are sent as they are produced, so the user starts reading at roughly one second while generation continues underneath.

And it is why the meaningful latency metrics are a pair rather than a single number:

• Time to first token (TTFT) — how long until something appears. This is the number users experience as responsiveness. Target: under a second.
• Time per output token (TPOT) — the pace afterwards. It only needs to beat reading speed, so roughly 30 to 50 tokens per second is sufficient.

Optimizing them pulls in opposite directions, which is the central tension of the rest of this lesson.

Step two: prefill and decode are different workloads

Generating a response has two phases with completely different performance characteristics.

Why decoding is memory-bound — the single most important fact in LLM serving

This is unintuitive and it explains nearly every serving optimization, so it is worth deriving.

To generate one token, the model must perform a forward pass. A forward pass touches every parameter in the model. Those parameters live in GPU memory and must be read into the compute units.

Weights to read per token = 70B params x 2 bytes = 140 GB
Memory bandwidth          = 3,350 GB/s
Maximum token rate        = 3,350 / 140 = 23.9 tokens/second

Twenty-four tokens per second is the ceiling for a single sequence — not because the arithmetic is slow, but because you cannot pull 140 GB from memory any faster.

And here is the waste: with one user, that entire 140 GB read produces exactly one token. The GPU's arithmetic units are almost entirely idle, waiting on memory.

Now the crucial observation. If you process many sequences simultaneously, you read those 140 GB of weights once and use them for every sequence in the batch:

Throughput scales nearly linearly with batch size while per-user speed stays constant. That is a remarkable property and it has no analogue in conventional serving, where adding concurrent requests to a saturated server makes everyone slower.

The reason it works is that decode is bandwidth-bound and the batch is not — the weights were going to be read regardless, so extra sequences ride along for free until the arithmetic units themselves saturate.

This is why batching is not an optimization here. It is the difference between a viable service and an impossible one.

Step three: the KV cache, and why it bounds everything

If large batches are free throughput, why not batch a thousand sequences? Because each one occupies memory that grows with the conversation.

The KV cache is the reason context length is expensive

Recall from Lesson 1 that generating token N requires attending to tokens 1 through N-1. Recomputing that from scratch each step would be quadratic and hopeless, so the intermediate key and value tensors for every previous token are kept in GPU memory. That store is the KV cache.

For a 70B-class model with 80 layers and a model dimension of 8,192:

KV per token = 2 (keys and values) x 80 layers x 8,192 x 2 bytes = 2.62 MB

Per token. Per sequence. So:

A single sequence at 128k context needs more memory than four GPUs, purely for its cache.

Modern models reduce this substantially with grouped-query attention, which shares key and value tensors across groups of attention heads. At 8 KV heads for 64 query heads the cache drops eightfold — 4,000 tokens of context falls from 10.5 GB to about 1.3 GB.

Now do the batching arithmetic honestly. Two 80 GB GPUs give 160 GB. The weights take 140 GB. That leaves 20 GB for KV cache:

20 GB / 1.3 GB per sequence = about 15 concurrent sequences

Fifteen. Not the 256 the throughput table wanted.

This is the real constraint, and it is why serving stacks are memory-allocation systems as much as inference systems. The fixes are all about buying back KV space: add GPUs so weights are a smaller fraction of total memory, quantize the weights, quantize the cache itself, page it in fixed blocks so fragmentation does not waste space, or simply serve shorter contexts.

It also gives a hard-nosed answer to a product question. "Why does a longer context window cost more?" Not because of a pricing decision — because KV cache scales linearly with context length and directly displaces the concurrency that pays for the hardware.

Step four: size the fleet

Aggregate demand    = 21.7 M tokens/second
Per-GPU throughput  = ~2,000 tokens/second (batched, mid-size model)
GPUs required       = 21.7M / 2,000 = about 10,900 GPUs

Put that beside 0.27 web servers

The previous lesson's corrected estimate said the web tier needs less than one server. The inference tier needs roughly ten thousand GPUs — and at data-centre prices that is capital in the billions before electricity.

Two things follow, and both are worth saying out loud in an interview.

First, the ratio between the two numbers is the design. Every architectural decision in this chapter should be evaluated by whether it moves work from the ten-thousand-GPU side to the less-than-one-server side. Semantic caching does. Routing simple prompts to a small model does. Adding a feature that requires an extra model call does the opposite, and needs to justify itself against roughly a dollar-per-thousand-requests of hardware.

Second, doubling per-GPU throughput halves the fleet. Look at the table: the difference between 1,000 and 4,000 tokens per second per GPU is 16,000 GPUs. That is why the batching and KV-cache mechanics above are not implementation trivia — a scheduling change that raises average batch size is worth more than any amount of application-tier optimization.

This is the inverted priority that defines AI systems. In every previous chapter, the expensive tier was storage or bandwidth and compute was noise. Here compute is four orders of magnitude more expensive than everything else combined, so effort belongs there and nowhere else.

Step five: the scheduling problem

Fixed batching wastes enormously here, for a reason specific to generation: sequences in a batch finish at different times. One user asks for a word, another for an essay. With static batching the whole batch waits for the longest.

Continuous batching — evict and admit at every step

The fix is to treat the batch as a living set rather than a fixed group. At every decoding step, completed sequences are evicted and waiting requests are admitted into the freed slots.

The scheduler is deciding, thousands of times a second, how many sequences to run given remaining KV memory — and it faces the tension from the start of this lesson directly:

• Prefill is compute-heavy and produces the first token. Prioritizing it improves TTFT for new arrivals.
• Decode is bandwidth-bound and continues existing responses. Prioritizing it improves TPOT for users already reading.

They contend for the same hardware. Admit too many new requests and existing users watch their text stutter; admit too few and new users stare at a blank screen. Production schedulers mix the two deliberately, and this trade-off is worth naming — it is the clearest example in the chapter of a genuine engineering choice with no free answer.

How this changes the standard scaling answers

Several reflexes from earlier chapters behave differently here, and knowing which ones is a good marker of having thought about it.

Autoscaling is slow. Adding a web server takes seconds. Adding an inference server means loading 140 GB of weights into GPU memory — minutes, not seconds. So capacity must be provisioned ahead of demand, and the smooth reactive scaling assumed elsewhere does not apply.

Load balancing is not round-robin. Requests are wildly unequal — a 50-token reply and a 2,000-token reply cost forty times as much and neither is knowable in advance. Sensible routing tracks KV memory occupancy and active sequence count per server, not request counts.

Queuing is normal, not a failure. Because GPU capacity is fixed and expensive, running at high utilization is the point. A short queue is a sign of correct provisioning. What matters is that queue time is counted inside TTFT, since the user cannot tell waiting from thinking.

The failure mode is memory, not CPU. A server does not slow down gradually as it fills — it runs at full speed until KV cache is exhausted, then must either refuse admissions or evict a sequence mid-generation. Admission control is therefore load-bearing rather than defensive.

Interview signal by level

Next: the building blocks, and the high-level flow.