Building Blocks We Will Use

In one line: four of these five blocks do exactly what they did in earlier chapters. One does not, and spotting which is the point of the lesson.

Based on our requirements and estimations, we will use the following core components.

Additional components, such as an API gateway and ZooKeeper, are also critical.

The cache is the block that means something different here

Four of these behave exactly as the building-block chapters described. The cache does not, and the difference is large enough that reusing the word without qualification is misleading.

Everywhere else in this course, a cache stores a copy of data that exists somewhere else. The distributed cache chapter's whole framing was avoiding a slower read of the same bytes. A cache hit and a cache miss return identical results; the only difference is latency.

Here there are two distinct caches doing two different jobs:

Recent conversation context (Redis). This is a conventional cache — the turns exist in the database too, and Redis just avoids the round trip. Standard.

Generated responses. This one is not conventional at all. The response does not exist anywhere else; it was produced by burning GPU time. Caching it is not avoiding a read, it is avoiding a computation that costs roughly a million times more than the read would.

That inverts the economics of caching completely. In the distributed cache chapter, a miss cost you a few milliseconds of database latency. Here a miss costs seconds of GPU occupancy on hardware in the ten-thousand-unit fleet from the last lesson. Which means:

• A far lower hit rate is still worth having. Even 5% is enormous when each hit saves a GPU-second.
• Approximate matching becomes worthwhile — which is exactly why the next lesson reaches for a semantic cache rather than an exact-key one.
• Correctness gets harder, because a hit and a miss no longer return the same thing.

When the cached item is expensive to produce rather than merely slow to fetch, every caching rule you learned changes. That is the single most useful transfer from the classic building blocks to this domain.

The CDN is here for what the model does not produce

Easy to skim past, but the detailed design places it deliberately: the CDN serves UI assets — scripts, styles, fonts, images — while "generated responses from the model are handled separately."

That separation is the point. The CDN cannot cache a response, because responses are unique per user and streamed as they are produced. Its entire job is keeping everything that is not generation off the expensive path.

Compare that building block, where the CDN carried the product itself — the video bytes — and was the dominant infrastructure. Here it carries the wrapper around the product and is almost incidental.

That contrast is a good summary of the whole chapter. The valuable payload has moved from bytes that can be replicated to computation that cannot.

ZooKeeper coordinating model servers has a twist worth naming

ZooKeeper appears where you would expect — "tracking active instances, distributing workloads, and handling failovers." Same role as the primary-replica mapping in that building block or segment assignment in Google Maps.

The twist is how slow recovery is. Every earlier chapter could treat replacing a failed node as fast: start a process, let it warm up, resume serving.

Here, bringing a model server into service means loading 140 GB of weights into GPU memory. That is minutes. So the coordination layer cannot rely on reactive replacement — it needs warm standby capacity already holding the weights, which is expensive precisely because that hardware is the expensive kind.

It also means failure is costly for in-flight work. A web request lost to a crash is retried in milliseconds. A sequence that has generated 800 tokens and loses its server has lost 800 forward passes of accumulated KV cache, and must start over.

Next: the high-level design and the path a prompt takes through it.