Resource Estimation

In one line: this lesson contains a genuine mistake in the published source. Working out why it is wrong is more instructive than any correct calculation in the chapter, because the correct answer shows that the whole estimation is measuring the wrong resource.

Assumptions

• Users: 1 billion total users, with 150 million daily active users (DAU)
• Requests: each active user sends ~10 requests daily, totaling 1.5 billion requests per day
• Request size: approximately 2 KB (mostly short text prompts)
• Response size: approximately 5 KB (AI-generated replies are typically longer)

Ten requests per user per day is a conversation, not ten conversations

Worth pausing on, because it changes what the system is serving.

A "request" here is one turn. Ten turns is roughly one or two conversations with follow-ups — which matches the Harry Potter exchange from Lesson 1.

The consequence is that requests arrive in bursts with strong locality: ten turns from one user within a few minutes, all sharing conversation history. That is excellent news for caching, and it is precisely why Redis holds recent conversation context. Ten independent requests spread across a day would make that cache nearly useless.

It also means sessions are the natural unit of affinity. Routing a user's turns to the same application server keeps their context warm — the same argument that building block made for session stickiness, arriving here for a different reason.

Requests per second

Total daily requests = 150,000,000 x 10        = 1.5 billion
RPS                  = 1,500,000,000 / 86,400  = 17,361 requests/second

That figure of 17,361 RPS is used correctly throughout the bandwidth calculations. Keep it in mind for the next section.

Number of servers — the published calculation

The formula:

Servers needed at peak load = (Number of requests/second) / (RPS of server)

Then, using a standard web server capacity of 64,000 RPS, it computes:

Servers needed at peak load = 150 million / 64,000 = 2,343 servers

This is wrong by a factor of 8,640 — and the design contradicts itself one sentence earlier

The formula says requests per second divided by server RPS. The calculation substitutes 150 million, which is the daily active user count, not a rate. The units do not match: users divided by requests-per-second is not a server count.

Substituting correctly:

Servers = 17,361 RPS / 64,000 RPS-per-server = 0.27 servers

Less than one server. The error factor is exactly 86,400 / 10 = 8,640 — the ratio between DAU and RPS given ten requests per user per day.

The strongest evidence that this is a slip rather than a modelling choice is that the design says so itself, in the sentence immediately before the calculation:

"Using a standard web server capacity of 64,000 RPS, the web tier requirements are minimal. However, this calculation applies to application servers handling connections, not the GPU servers required for inference."

That is exactly right — and then it computes 2,343 servers anyway. The prose and the arithmetic disagree, and the prose is correct.

Do not carry 2,343 into an interview. If you meet this figure in study material, the recoverable insight is not the number but the check: always confirm that the quantity you substitute has the units the formula asks for. A rate divided by a rate gives a count; a population divided by a rate gives nonsense.

But 0.27 servers is not the answer either — it is the point

Do not now claim the system needs one server. That figure is only what it appears to be for a trivial reason: the classic web-server estimate models the wrong work.

The 64,000 RPS figure assumes a server doing what web servers do — parse a request, look something up, serialize a response, in a millisecond or two. Every previous chapter in this course used it validly, because that genuinely was the work.

Here, the work is running a forward pass through a network with hundreds of billions of parameters, a thousand times over, on a GPU. No amount of 64,000-RPS arithmetic captures that.

So the honest reading of the corrected number is:

The web tier is a rounding error. The entire cost of this system is in a resource the estimation never mentions.

Which is precisely what the design's own note concedes, and then declines to compute:

"Generative AI relies heavily on GPU servers for inference. For GPU requirements, refer to the back-of-the-envelope calculations for model training."

The next lesson does that calculation, because it is the only estimation in this chapter that determines anything.

Storage

Total storage per day = 1.5 billion requests x 2 KB = 3 TB/day
Total storage per year = 3 TB x 365                 = 1.1 PB/year

Three orders of magnitude smaller than any previous chapter

For scale: YouTube's chapter estimated ~9.5 PB of duplicate uploads alone, Quora 42 PB per year, and Google Maps about 20 PB of near-static map data.

ChatGPT at 150 million DAU stores 1.1 PB per year — and that is generous, since it counts only prompts and would roughly triple by including responses.

Text is tiny. That is the whole reason. A 2 KB prompt is about a thousandth of a single 2 MB map tile, and a millionth of a video upload.

So storage, like bandwidth, is simply not a design constraint here. If you spend interview minutes designing a sharding strategy for conversation history, you are optimizing something that costs almost nothing. The estimation's job is to tell you where not to look, and it does that well — even though the section it needs is the one it skips.

One aside here is genuinely important:

Modern deep learning models are massive, requiring significant storage and compute power. For example, a model with 3 billion parameters in FP16 (16-bit floating point) format occupies around 6 GB. To enable real-time inference, these models are loaded into GPU memory, often distributed across multiple GPUs to balance the load.

The arithmetic behind 6 GB — and why it is the number that actually matters

3,000,000,000 parameters x 2 bytes (FP16) = 6,000,000,000 bytes = 6 GB

Parameter count times bytes per parameter. That is the whole formula, and it is worth committing to memory because it determines how many GPUs you need before a single user arrives.

Run it at realistic scale:

A current data-centre GPU holds roughly 80 GB. So a 175B model in FP16 needs at least five GPUs just to hold the weights, before any user data — which is exactly what "distributed across multiple GPUs" means.

Notice that this appears in the storage section, and it is the one storage number that matters — but it is not disk storage. It is GPU memory, a resource that is roughly a thousand times more expensive per byte and cannot be scaled by adding disks.

And note the lever the table exposes: quantization halves the requirement. Serving in INT8 rather than FP16 takes 175B from five GPUs to three, at some cost in output quality. That trade — precision against hardware count — has no analogue in any earlier chapter, and it is one of the highest-leverage decisions in an LLM serving design.

Bandwidth

Incoming = 17,361 req/s x 2 KB x 8 bits = 277.8 Mb/s
Outgoing = 17,361 req/s x 5 KB x 8 bits = 694.4 Mb/s
Total                                   ~ 972 Mb/s, under 1 Gb/s

Both published figures reproduce exactly.

Under 1 Gb/s — put that next to the rest of the course

ChatGPT serves 150 million daily users on less egress than a single well-provisioned rack. Maps moves 428 times more.

This is the clearest signal in the chapter. Bandwidth is not a problem. Storage is not a problem. Servers, in the classic sense, are not a problem. Every quantity this estimation computes is negligible.

What that tells you is not that the system is easy — it is that the expensive resource is one the standard estimation template does not have a row for. The template was built for systems that move bytes. This system burns FLOPs.

That is a transferable lesson beyond LLMs. When every number in your estimate comes back trivially small for a system you know is expensive to run, the model is wrong, not the system. Find the resource you are not counting.

One useful nuance on the egress figure: 694 Mb/s is the aggregate byte rate, but responses are streamed token by token rather than delivered as 5 KB blocks. So the connection stays open for the several seconds generation takes. The bytes are trivial; the concurrent connections are not — the same C10K shape that building block met with WebSockets, arriving here for a different reason.

Interview signal by level

Next: the estimation the design declined to do.