Concept Drills: 15 ChatGPT Probes

Cover the answer, attempt it out loud, then compare. If your answer would apply equally well to a web application, it is not yet an answer about this system.

1. What kind of problem is this?

Weak: "A large-scale distributed system with an AI model in it."

Strong: "A compute problem, and it's the first one in the course. Run the standard template and everything comes back negligible — 1.1 PB a year of storage against Quora's 42 PB, under 1 Gb/s of bandwidth against YouTube's 12 Tb/s, and less than one web server at 64,000 RPS.

That's the diagnostic. When every number in your estimate is trivial for a system you know costs billions to run, the model is wrong, not the system. The template counts bytes moved; this system burns FLOPs. The resource that matters — GPU inference — has no row in the template at all."

2. Why can't a single response be made faster by adding GPUs?

Weak: "Because the model is too big to parallelize."

Strong: "Because generation is autoregressive. To produce token 50 the model must already have produced tokens 1 through 49 — each output is fed back as input. A 1,250-token response is 1,250 strictly sequential forward passes.

More GPUs make a response possible — a 175B model in FP16 needs 350 GB and won't fit on one device — and they let you serve more responses concurrently. They don't shorten the dependency chain of a single one.

It's the same shape as Dijkstra in the Maps chapter: an inherently sequential algorithm meeting a latency budget. There the answer was precomputation; here it's streaming — you can't make the response finish sooner, so you make it start sooner."

3. Why is decoding memory-bandwidth bound rather than compute bound?

Weak: "Because GPUs have limited memory."

Strong: "Not capacity — bandwidth. Every token requires a forward pass touching every parameter, which means reading all the weights from GPU memory into the compute units.

70B params x 2 bytes = 140 GB read per token
3,350 GB/s bandwidth / 140 GB = about 24 tokens/second

Twenty-four tokens per second is the ceiling for one sequence — not because the arithmetic is slow but because you can't move 140 GB any faster. With a single user, that entire read produces one token and the arithmetic units sit almost idle.

That idleness is the opportunity, and it's what makes batching work."

4. Why does batching help so much here when it doesn't in web serving?

Weak: "It amortizes overhead across requests."

Strong: "Because the bottleneck is a read that happens regardless of batch size. Process 128 sequences together and you read those 140 GB once and use them for all 128.

Throughput scales with batch size while per-user speed stays constant. That's the opposite of conventional serving, where adding concurrency to a saturated server slows everyone down.

So batching isn't an optimization here. It's the difference between an 11,000-GPU fleet and an impossible one."

5. Then why not batch a thousand sequences?

Weak: "Diminishing returns."

Strong: "KV cache memory. Each sequence keeps key and value tensors for every token it has seen, so attention doesn't have to recompute them — about 2.6 MB per token for a 70B-class model, or 1.3 GB for 4,000 tokens with grouped-query attention.

Do the honest arithmetic: two 80 GB GPUs give 160 GB, weights take 140, leaving 20 GB. That's about fifteen concurrent sequences, not 256.

Which is why an inference stack is a memory allocator as much as an inference engine — paged KV cache to stop fragmentation wasting space, quantized weights to free room, and continuous batching that evicts finished sequences and admits new ones every step.

It also explains a product fact cleanly: longer context costs more because KV cache scales linearly with context and directly displaces the concurrency that pays for the hardware."

6. What is the difference between prefill and decode, and why does it matter?

Weak: "Prefill reads the prompt and decode writes the answer."

Strong: "They're different workloads on the same hardware.

Prefill processes the whole prompt in one pass, all tokens in parallel. It's compute-bound and it determines time to first token.

Decode generates output one token at a time, strictly sequentially. It's memory-bandwidth bound and it determines time per output token.

That split matters because the two metrics pull against each other in the scheduler. Prioritize prefill and new arrivals see text sooner while existing users' output stutters. Prioritize decode and current readers are smooth while new users stare at a blank screen. They contend for the same GPU, and there's no free answer — production schedulers deliberately mix them."

7. What's wrong with the published server estimate of 2,343?

Weak: "It's a bit optimistic."

Strong: "It's wrong by a factor of 8,640, and it's a units error. The formula is requests per second divided by server RPS, but the calculation substitutes 150 million — the daily active user count, not a rate. Users divided by requests-per-second isn't a server count.

Substituting correctly: 17,361 / 64,000 = 0.27 servers.

The giveaway is that the source says so itself one sentence earlier — 'the web tier requirements are minimal' — and then computes 2,343 anyway. The prose is right and the arithmetic isn't.

The lesson isn't the number, it's the check: confirm the quantity you substitute has the units the formula asks for. And the corrected answer is the real finding — the web tier is a rounding error, and the entire cost is in a resource the estimation never mentions."

8. Why does exact-match caching fail for generated responses?

Weak: "Because prompts vary too much to match."

Strong: "It fails in both directions at once, which is the tell that the key is wrong rather than the policy.

False misses: 'How do I reverse a linked list in Python?' and 'python code to reverse a linked list' are three separate keys wanting one answer — three times the GPU cost.

False hits: two users both send 'summarize the document.' Byte-identical, and they must not collide, because the prompt text isn't the whole input — conversation history and profile are inputs too, and the key can't see them.

Missing and colliding simultaneously isn't a tuning problem. It means the key should be meaning, which is what the semantic cache provides."

9. What's the risk in a semantic cache, and how do you contain it?

Weak: "It might return a slightly outdated response."

Strong: "Much worse than outdated. A false hit is a fabricated answer, not a stale one.

'What is the capital of Australia?'  -> Canberra
'What is the capital of Austria?'    -> embeds very close

One letter apart, vectors close together, and returning 'Canberra' for Austria is a confidently wrong answer the user has no way to detect. In a normal cache a bad hit gives you old truth; here it gives you invented truth.

Three containments: scope the cache to first-turn prompts with no history or personalization, where a shared answer is legitimately correct; embed the assembled context rather than the bare prompt, so 'summarize the document' from two users produces different vectors; and make the threshold per-domain, high for factual lookups and lower for open-ended creative requests.

Note the approximate nearest neighbour search itself is safe — a missed match just means we generate, which is what we'd have done anyway. Design approximations so they fail in the free direction."

10. Why is even a 5% cache hit rate worth having?

Weak: "Every bit helps."

Strong: "Because the hit-to-miss cost ratio is unlike anything in conventional caching. A hit is an embedding plus an ANN lookup — a few milliseconds, no GPU. A miss is prefill plus 1,250 decode steps — seconds of GPU occupancy.

So 5% of traffic avoided is roughly 5% of an 11,000-GPU fleet — hundreds of GPUs, real capital.

The general principle: when the cached item is expensive to produce rather than merely slow to fetch, low hit rates start paying for themselves. A 5% hit rate would be embarrassing for a web cache and is valuable here."

11. What is prefix caching and why is it different from semantic caching?

Weak: "Another name for the same thing."

Strong: "Different object and different risk profile.

Semantic caching reuses whole responses by approximate meaning. Prefix caching reuses intermediate KV state for a prefix that is byte-identical.

The prefixes are everywhere: the system prompt is the same across every request, and within a conversation turn five contains turns one through four verbatim. Their KV entries are identical each time, so compute once and reuse.

The crucial difference is that prefix caching is exactly correct — no threshold, no false hits, no accuracy risk — while the semantic cache is approximate and can fabricate.

It also cuts prefill, so it improves time to first token, the metric users actually feel. And it's why multi-turn conversations don't get linearly slower as they grow despite the context growing every turn. If you can only name one cache, name this one."

12. Why is moderation after generation, and is that sufficient?

Weak: "So we check the response before sending it."

Strong: "After, because output is what reaches the user and input doesn't predict it — 'write a realistic villain's monologue' is benign and can produce unpleasant text, injection hides behind innocuous phrasing, and sampling means you can't infer this response from the prompt.

But 'after instead of before' is the wrong framing. It should be both, and the argument for input screening is cost: rejecting a clearly disallowed prompt with a millisecond classifier avoids seconds of GPU time. Same principle as the semantic cache — anything that avoids invoking the model pays for itself.

And the placement has a problem with no clean answer: moderation wants a complete response and streaming means there isn't one. Buffer the whole thing and you destroy the responsiveness streaming existed for. Stream and retract and the user already read it. Moderate in windows and you catch local harm but miss passages innocuous sentence-by-sentence and unacceptable as a whole.

Layer all three and say plainly that none is sufficient. Also: moderation checks policy, not truth."

13. Is more retrieved context better?

Weak: "Generally yes, more information helps the model."

Strong: "No — it degrades in three distinct ways.

Cost and latency: more chunks means more prefill tokens on every request and more KV cache, so fewer concurrent users.

Attention dilution: models attend less reliably to material in the middle of a long context. Burying the one relevant chunk among nineteen irrelevant ones can make it less likely to be used.

Contradiction: retrieve enough and some sources disagree — an old policy and its replacement. The model can't reliably adjudicate and may confidently follow the wrong one. Retrieving more raises the chance of retrieving something wrong.

So the fix is ordering, not volume: rerank with a more expensive scorer, keep the top few, place the most relevant material at the edges rather than the middle, prefer recency on conflict.

Precision matters more than recall here — the opposite of a search engine, where a user can ignore result seven. The model reads everything you give it."

14. What does retrieval buy that a bigger model can't?

Weak: "Access to more information."

Strong: "Three things, and the third is the architectural one.

Knowledge that was never in training — your internal documents, anything past the cutoff. No model size fixes absent data.

Attribution. A retrieved chunk has a source, so the answer can cite it and a user can check. A claim from weights has no provenance. That's the closest this design comes to an accuracy answer — it makes claims checkable, not correct.

Updates without retraining. Fixing a fact means re-embedding one chunk — minutes. Fixing knowledge baked into weights means retraining — weeks.

That last one cleanly separates capability, which lives in the weights and changes slowly, from knowledge, which lives in the index and changes continuously. Structurally identical to the Maps chapter's base-versus-transitory edge weights: precompute the stable part, keep the volatile part in a layer you can change cheaply."

15. What can this design not do?

Weak: "It could struggle at very high scale."

Strong: "It cannot tell whether an answer is true.

Correctness isn't in the requirements, moderation explicitly checks policy rather than truth, the cache can't know, and no other component looks. The model can produce fluent, confident, well-formatted falsehood and every component will pass it through.

Retrieval with citations helps by making claims checkable, but that's a mitigation, not a fix.

Two related absences. Cost isn't a listed requirement, despite driving model size, quantization, batching, and cache thresholds more than anything that is listed. And evaluation appears nowhere — the feedback loop closes in weeks through retraining, and without a held-out gate you ship regressions, since a model can be better on the specific complaints and worse overall.

Worse, feedback is biased toward satisfaction rather than correctness: a confident wrong answer gets a thumbs-up, a correct 'I don't know' gets a thumbs-down. Train on that naively and you optimize for plausibility — precisely the wrong direction for a system with no accuracy safeguard. An optimization loop drives whatever it measures, and here the measurement and the goal diverge exactly where it matters most."