What Changes When You Estimate an AI System
In one line: the estimation habits from classic system design still work, and three things move — the unit you count, the resource that runs out first, and whether cost is worth mentioning at all.
Three shifts
The unit is the token, not the request
A classic service is sized in requests per second, because requests are roughly interchangeable. Two GET calls to the same endpoint cost about the same.
A generative request is not interchangeable with another one. A three-word question and a forty-page document summary arrive through the same endpoint and differ in cost by three orders of magnitude. So the unit that predicts anything is the token, and requests per second is nearly meaningless on its own.
The binding resource is memory, not compute
In a classic service the usual first ceiling is CPU. In LLM serving it is almost always GPU memory, and specifically the KV cache — the per-request state that grows with conversation length.
This has a consequence that surprises people: your concurrency limit depends on how long the conversations are, not just how many there are. A model that serves 64 concurrent short chats may serve 4 concurrent long-document sessions on identical hardware. Capacity is a function of context length, and that gets its own lesson.
Cost is a first-order design constraint
For a classic service, serving cost is usually small compared to engineering cost, which is why candidates are used to omitting it.
For a generative system it can dominate the product economics entirely. A support bot at ten cents a conversation and a million conversations a month is 100,000 dollars a month, and that number decides whether the feature ships. An estimation answer that never produces a cost figure is incomplete in this round in a way it was not before.
| Question | Classic system | AI system |
|---|---|---|
| What do you count? | Requests per second | Tokens per second, split into prompt and completion |
| What runs out first? | CPU, then IO | GPU memory — weights plus KV cache |
| What sets concurrency? | Threads and connection pools | Free GPU memory divided by per-request cache |
| Does cost matter? | Usually a rounding error | Often the constraint that decides the product |
| What is elastic? | Add commodity servers in minutes | Accelerators are scarce, slow to acquire, and reserved |
The last row is worth a sentence in an interview. You cannot autoscale a GPU fleet the way you autoscale web servers — capacity is often contracted months ahead, which turns a capacity estimate into a procurement commitment rather than a config change.
What has not changed
Everything else. Traffic estimation from daily active users, peak-to-average ratios, storage growth, replication multipliers, bandwidth — all of it applies unchanged, and this chapter assumes it rather than re-deriving it.
The AI-specific arithmetic sits on top of a classic estimate. You still work out how many requests arrive at peak; you then work out what each one costs in tokens, memory and money.
What estimation is for
Worth restating because candidates routinely misjudge it: the goal is not a precise number. It is to eliminate options.
An estimate earns its place when it changes a decision. "One hundred million items at a millisecond each is 100,000 seconds against a 30ms budget" does not need to be accurate to two significant figures — it needs to be right about the order of magnitude, because six orders of magnitude means the design must change and no amount of tuning helps.
Key takeaway
Classic capacity planning still applies and this chapter builds on it. Three things change: the unit becomes the token because two requests to one endpoint can differ in cost a thousandfold, the binding resource becomes GPU memory so concurrency depends on context length, and cost becomes a first-order constraint that can decide whether the product exists. Estimate to eliminate options, round aggressively, and only do a calculation out loud when it changes what you draw next.
Next: the numbers worth committing to memory.