Resource Estimation

In one line: the server number is wrong in the module's usual way, and buried inside the same section is the single most important architectural fact in the chapter.

Assumptions

Note what the chain multiplies out to: tokens, not requests. That is the unit this system is actually sized and billed in.

Registered users        500 million
Daily active users       50 million
Queries per user/day              5
Total requests/day      250 million
Request size                  ~2 KB   (metadata, headers, user text)
Response size                 ~4 KB   (LLM output is longer than input)

Servers

We assume that all requests from daily active users (50 million) arrive at once to estimate peak load. Under this worst-case scenario, we consider 50 million requests per second.

50,000,000 / 64,000 ≈ 782 servers

DAU-as-RPS again — and this time it is inconsistent with its own stated premise

The convention is familiar by now. What is new is that the chapter states the premise explicitly and then does not follow it.

The premise is "all requests from daily active users arrive at once." There are 250 million requests per day, not 50 million. So even granting the absurd assumption, the number should be 250 million — the chapter substitutes daily active users where its own worst case calls for total requests.

Actual average:  250M / 86,400        =  2,894 RPS  ->  0.045 servers
At a 3x peak:                            8,681 RPS  ->  0.136 servers
Published:                              50,000,000  ->    782 servers
                                                        -----------
Error factor:                                            17,280x

One machine, comfortably. And 2,894 RPS is not an unusual figure — it is roughly what a single well-tuned application server handles.

This is the ninth appearance of the convention in the module and its most self-contradictory instance: the section defines a worst case, computes a different quantity, and reports it as the worst case.

When a section states its assumption, check that the number it produces follows from that assumption. The same discipline caught that building block's "we use the peak traffic load", that building block's identical phrasing, and that building block's 4-hour SLA.

GPU servers

However, GPU inference servers are the real bottleneck. Assuming each GPU server handles approximately 200 inference requests per second:

50,000,000 / 200 = 250,000 GPU servers

Practical tip: In practice this number would be significantly lower... the infrastructure requirement would likely scale to hundreds or a few thousand GPU servers.

'GPU inference servers are the real bottleneck' is the best sentence in the estimation

Set the arithmetic aside for a moment, because this observation is correct and it is the one that matters.

The chapter gives two capacity figures:

Application server:  64,000 requests/second
GPU inference server:   200 requests/second
                     ---------------------
Ratio:                      320x

One tier of this system is 320 times weaker than the tier in front of it. That single fact explains the entire architecture:

Four optimizations, one goal. When one tier is hundreds of times weaker than the one in front of it, the architecture becomes a filter, and every component before the model exists to reduce what reaches it.

It is also why the 782-server figure is doubly beside the point: application servers were never the constraint. Sizing them precisely and ignoring the GPU tier is optimizing the part that is not scarce.

The source doubts its own number — and its correction is still 5 to 46 times too high

This is only the second source in the module to flag its own estimate as unrealistic (the first was the typeahead chapter), and that instinct is worth crediting.

But work the corrected figure through:

Published:        50,000,000 / 200  =  250,000 GPU servers
Source's own fix: "hundreds or a few thousand"
Actual average:        2,894 / 200  =     14.5 GPU servers
At a 3x peak:          8,681 / 200  =     43.4 GPU servers

Roughly 15 GPU servers at average load, 43 at a threefold peak. The source's own revised range of "hundreds to a few thousand" is still 5× to 46× too high, because it corrects the reasoning ("traffic is distributed over time") without correcting the input — the estimate is redone qualitatively rather than recomputed.

A correction that fixes the argument and not the arithmetic leaves the number wrong. The typeahead chapter made the same move: it doubted its estimate, then substituted a different quantity rather than recomputing from the right one.

The honest version, and a good thing to say in an interview:

"At 250 million requests a day that's about 2,900 requests per second average, maybe 8,700 at peak. At 200 inference requests per GPU server, that's roughly 15 to 45 GPU servers — and I'd size up from there for redundancy, regional distribution, and headroom, not down from a worst case that assumes a day's traffic in one second."

200 requests/second per GPU is itself a shaky number, and Lesson 3 explains why

The figure is stated without justification, and it hides the thing that actually determines GPU throughput: an inference request is not a unit of work.

"Store hours?"          -> ~20 output tokens
"Troubleshoot my
 device not charging"   -> ~800 output tokens

Those differ by 40× in generation cost, because generation is sequential — the model produces one token at a time and each token requires a forward pass.

So "200 requests per second" is meaningless without an assumed output length, and the right capacity unit for a GPU tier is tokens per second, not requests per second.

BAD:   requests/second   -> varies 40x with response length
GOOD:  tokens/second     -> the actual work the hardware does

That reframing also explains why the design's optimizations work: shorter answers, cached answers, and small-model answers all reduce tokens generated, which is the quantity the GPU tier is actually rate-limited on.

Size a GPU tier in tokens per second and convert to requests using an assumed output distribution — never the reverse.

Storage

(2 KB request + 4 KB response) x 250M/day  = 1.5 TB/day
x 365                                       = 548 TB/year
1-trillion-parameter model in FP16          =   2 TB  (fixed)

All three reproduce exactly.

The model-storage figure is small and the arithmetic behind it is worth knowing

1 trillion parameters × 2 bytes (FP16) = 2 TB is correct, and the general form is the useful thing:

Model size in bytes = parameters x bytes per parameter

FP32  4 bytes/param    1T params ->  4 TB
FP16  2 bytes/param    1T params ->  2 TB     <- the source's figure
INT8  1 byte /param    1T params ->  1 TB
INT4  0.5 bytes/param  1T params -> 500 GB

The reason this matters is not disk cost — 2 TB is trivial. It is GPU memory, because the weights must be resident to serve:

2 TB of weights / 80 GB per high-end GPU = ~25 GPUs, just to HOLD the model
  -> before any KV cache for in-flight requests
  -> before any batching headroom

Model storage is a disk footnote and a GPU-memory constraint, and it is the second reason the GPU tier is the bottleneck: a large model does not fit on one accelerator, so serving it means sharding across several and paying the interconnect cost on every token.

Quantization is the lever — INT8 halves the memory and roughly doubles the achievable batch size, at some accuracy cost. That is the trade behind the design's "lightweight model" tier.

548 TB/year of conversation logs is not a storage problem, it is a privacy problem

The volume is unremarkable — that building block moved more in ten days. What makes it notable is what it contains.

The requirements say conversations "often contain sensitive personal or financial data" and name GDPR. So this is 548 TB/year of:

order numbers · addresses · payment disputes · account details
· whatever a frustrated customer types into a text box

Three consequences the estimation does not draw:

Retention has a ceiling, not a floor. Unlike the payment chapter, where regulation demanded long retention, here regulation demands the opposite — data minimization. Keeping conversation logs indefinitely is a liability.

Deletion must be supported, which means logs need to be keyed by user and purgeable — the GDPR problem that building block met, with the same answer available (crypto-shredding or per-user partitioning).

The training use conflicts with the privacy use. The design says feedback and logs feed "model retraining", and training on raw customer conversations means personal data entering model weights, from which it cannot be deleted.

When the data is conversations, the storage estimate is the smallest of the questions it raises.

Bandwidth

50 M/s × 2 KB = 100 GB/s incoming · 50 M/s × 4 KB = 200 GB/s outgoing

Inherits the 17,280x error, and the real figures are small enough to be uninteresting

The arithmetic is right given the wrong rate. Corrected:

Published:  100 GB/s in  (800 Gbps),  200 GB/s out
Actual:     2,894 RPS x 2 KB =  5.8 MB/s in
            2,894 RPS x 4 KB = 11.6 MB/s out

Under 20 MB/s combined. Bandwidth is not a design consideration in this system at any scale you will be asked about — text is small, and there is no media.

Which is the useful conclusion: in an LLM system, bandwidth is never the constraint and compute always is. The bytes are trivial; the work per byte is enormous. That is the inverse of that building block, where the bytes were the whole problem.

One thing worth noting about the response-size assumption: 4 KB is a large support answer — roughly a thousand tokens, or three paragraphs. Most support responses are shorter, and since generation cost scales with output length, assuming long responses inflates the GPU estimate more than it inflates the bandwidth estimate.

The building blocks, and the two that carry the design

Load balancers · database (session context) · cache ("frequent query-response pairs to reduce redundant LLM inference calls") · pub-sub (async logging, feedback, analytics) · rate limiter.

The cache and the rate limiter are the two doing real work, and both for the same reason as everything else: protecting the GPU tier.

The cache's stated purpose — "reduce redundant LLM inference calls, directly supporting cost-aware routing" — is exactly right in intent. Lesson 6 shows that the mechanism implied (matching query-response pairs) has a poor hit rate on natural language, and that the components needed to fix it are already elsewhere in this design.

The rate limiter is more important here than in most systems in this module. Elsewhere it protects against abuse; here every request costs real money, so an unthrottled client is not merely load — it is an unbounded bill. When unit costs are high, rate limiting is a cost control before it is a security control.

Next: the latency budget, and why it cannot be met the way the chapter implies.