Resource Estimation
In one line: the error here is the most precisely diagnosable in the module. The assumption and the calculation are one line apart and disagree by exactly the ratio between them.
Assumptions
Active developers 5 million Requests per active hour 50 Active coding hours per day 4 Global regions 5 Total: 5M x 50 x 4 = 1,000,000,000 requests/day
The daily total is correct.
The per-developer rate is the number to hold onto
Convert the assumption into a rate, because the next section does not:
50 requests per active hour = 50 / 3,600 seconds = one request every 72 seconds = 0.0139 requests/second per active developer
One request every 72 seconds is a reasonable figure for inline completion with debouncing — the developer types, pauses, a request fires. It is not one per keystroke, because the client debounces (the design's own practical tip suggests 150–200 ms), and it is not one per minute either.
Hold onto 0.0139 requests/second per active developer. Everything below follows from it.
Servers
We estimate servers for peak traffic, and we assume that all users are active and making requests, which results in five million requests per second.
5,000,000 / 64,000 ≈ 79 servers
...if a single GPU server handles ~200 inference requests per second, at peak the system needs roughly
5,000,000 / 200 ≈ 25,000GPU servers.
The error is exactly 72x, and you can read it straight out of the assumption
The premise is "all users are active and making requests." Apply the chapter's own per-developer rate to all five million:
5,000,000 developers x 0.0139 requests/second = 69,444 RPS
The chapter says 5,000,000 RPS. The ratio is not approximate:
5,000,000 / 69,444 = 72
= 3,600 / 50
The assumption says one request every 72 seconds; the estimate assumes one request every second. That is the whole error, and it is stated outright in the bandwidth calculator, which lists "Average daily requests per user per second: 1."
Follow it through:
Published All devs active Realistic peak
(worst case) (1/3 of devs)
App servers 64,000 RPS 79 1.09 0.36
GPU servers 200 RPS 25,000 347 116
One application server covers the absolute worst case — every developer on the planet typing simultaneously. And the GPU tier, which is the one that matters, is roughly 70–120 servers at realistic regional peaks rather than 25,000.
Note the regional detail the chapter supplies and never uses: five global regions means activity is staggered by time zone, so the fraction of developers coding at any instant is closer to a fifth or a third than to all of them. The design has the information needed to compute a sensible peak and does not use it.
When a section states a per-user rate and then a total rate, divide one by the other and check the ratio is the number of users. Here it was 72× larger.
The GPU tier is the answer, and 347 is a genuinely large number
Correcting 25,000 to ~347 is not deflating the problem — it is locating it.
App servers: ~1 <- irrelevant, as in every LLM system GPU servers: 70-350 <- the entire cost and capacity story
A 320× capacity gap between the app tier and the GPU tier — the same ratio as the previous chapter — means the GPU count is the only server number worth discussing. Sizing app servers precisely while getting the GPU tier wrong by 72× is optimizing the part that is not scarce.
And the chapter's own note is well judged:
Caching frequently requested completions and applying model quantization can reduce the GPU server count by 30–40%.
Applied to the corrected figure, that is 347 → roughly 210, or 116 → 70. On the published figure it would be 25,000 → 15,000, which is the difference between two impossible numbers. The optimization advice is sound; it is attached to a baseline that makes it meaningless.
Optimizations quoted as percentages are only meaningful against a correct baseline.
Storage
If each request-response pair generates about 4 KB of telemetry, then at one billion requests per day the system produces roughly 4 TB/day.
1,000,000,000 x 4 KB = 4 TB/day ✅ (1.46 PB/year)
4 KB is exactly the request plus the response — so the telemetry is the design code
The bandwidth section, one paragraph later, assumes a 3 KB request and a 1 KB response.
Telemetry per event: 4 KB Request + response: 3 KB + 1 KB = 4 KB
Those are not independently derived numbers that happen to match. The telemetry is the prompt and the completion, stored verbatim.
That matters because of what a code assistant's prompt contains: the developer's current file, cursor context, open tabs, and retrieved repository snippets. So this system stores 1.46 petabytes per year of customer source code, and it sits directly against a stated non-functional requirement:
"Code context sent to the backend may contain proprietary source code... with options for enterprise customers to restrict data retention entirely."
The storage schema confirms the intent was smaller. It lists telemetry fields as "completion_id, session_id, timestamp, action, language, and latency metrics" — which is a few hundred bytes, not 4 KB. The schema describes metadata; the estimate prices code.
METADATA-ONLY telemetry: ~200 bytes -> 0.2 TB/day, no code retained CODE-INCLUSIVE telemetry: 4 KB -> 4 TB/day, 1.46 PB/year of customer source
A 20× difference in volume, and the entire difference is whether you keep the code.
The design needs both — the feedback loop wants prompts and completions for fine-tuning — so the honest resolution is tiering:
Metadata for ALL events -> always, cheap, no privacy exposure Full prompt/completion -> ONLY for tenants who opted in Enterprise "no retention" tenants -> metadata only, and say so
When a storage estimate is exactly the size of the payload, the system is storing the payload — and here the payload is the thing the privacy requirement was written about.
Bandwidth
5 M/s × 3 KB = 15 GB/sincoming ·5 M/s × 1 KB = 5 GB/soutgoing
Inherits the 72x error — and the asymmetry is the interesting part
Corrected:
Published: 15 GB/s in, 5 GB/s out All devs active: 208 MB/s in, 69 MB/s out Realistic peak: 69 MB/s in, 23 MB/s out
Modest, and bandwidth is not a constraint here — as in every LLM system, the bytes are trivial and the work per byte is enormous.
But the 3:1 input-to-output ratio is worth noticing, because it is the inverse of the support bot, and it explains where the cost goes.
SUPPORT BOT: 2 KB in, 4 KB out -> output-heavy -> DECODE dominates CODE ASSISTANT: 3 KB in, 1 KB out -> INPUT-heavy -> PREFILL dominates
A code completion is short — a few lines. The context is long — a file, open tabs, retrieved snippets.
That inversion is the single most important consequence of this estimate, and Lesson 3 develops it: because prefill scales with prompt length and prefill determines time-to-first-token, an input-heavy workload is a TTFT-bound workload. The support bot could hide its cost behind streaming; this system cannot, because its cost is incurred before the first token exists.
Read the input/output ratio to know whether you are prefill-bound or decode-bound — it determines which optimizations matter.
| Quantity | Published | Assessment |
|---|---|---|
| Requests/day | 1 billion | ✅ Correct |
| Peak RPS | 5,000,000 | 🔴 Exactly 72× high — the assumption says 1 request per 72 s; the estimate uses 1 per second. Worst case is 69,444 |
| App servers | 79 | 🔴 ~1 covers every developer typing at once |
| GPU servers | 25,000 | 🔴 ~347 at absolute peak, ~70–120 at realistic regional peaks — and this is the number that matters |
| Telemetry/day | 4 TB | ⚠️ Arithmetic correct — 4 KB = request + response exactly, so it stores source code (1.46 PB/yr) |
| Bandwidth | 15 / 5 GB/s | 🔴 Inherits the error; actual ~69 / 23 MB/s. The 3:1 input ratio is the useful signal |
| Cost | never computed | ⚠️ Stated as an NFR; the 30–40% optimization is quoted against an impossible baseline |
Two copy-paste artifacts, and what they suggest
Small, and worth noting because they point at how the estimate was produced.
The storage calculator is titled "Storage required for text-based requests on ChatGPT", and the bandwidth diagram is captioned "The total bandwidth required by Google Docs" — both carried over from other chapters.
That is consistent with the substantive error: the estimate was templated rather than derived. The template's shape is users × size ÷ time, with "users as requests per second" baked in — which is exactly the assumption that produces the 72× discrepancy, and exactly what the chapter's own per-hour rate contradicts.
A templated estimate inherits the template's assumptions, and the tell is when a stated assumption and a computed figure disagree by a clean ratio.
Key takeaway
The estimate is off by exactly 72×, and uniquely, you can read the error straight out of the assumptions: 50 requests per active hour is one every 72 seconds, while the calculation uses one per second — as the bandwidth calculator states outright. Even the absolute worst case is 69,444 RPS → about one app server and ~347 GPU servers, and with five staggered regions the realistic peak is 70–120 GPU servers. The GPU tier is the only server number worth discussing, and the chapter's sound 30–40% optimization advice is attached to a baseline that makes it meaningless — optimizations quoted as percentages need a correct baseline. Two further findings: the 4 KB telemetry figure is exactly the request plus response, so the system stores 1.46 PB/year of customer source code against a requirement promising retention controls; and the 3:1 input-to-output ratio inverts the support bot's, making this a prefill-bound, TTFT-bound workload.
Next: where the 300 milliseconds actually goes.