Building Blocks We Will Use

Why this matters: four of these five blocks do exactly what they did in earlier chapters. One does not, and spotting which is the point of the lesson.

Based on our requirements and estimations, we will use the following core components.

Building block	Role here
Load balancers	Distribute user requests across multiple servers and services
Databases	Store metadata and graph-structured data
CDNs	Deliver static content to end users, reducing latency and server load
Pub-sub system	Manage real-time messaging and events
Cache	Store frequently accessed data to improve performance

Additional components, such as an API gateway and ZooKeeper, are also critical.

The cache is the block that means something different here

Four of these behave exactly as the building-block chapters described. The cache does not, and the difference is large enough that reusing the word without qualification is misleading.

Everywhere else in this course, a cache stores a copy of data that exists somewhere else. The distributed cache chapter's whole framing was avoiding a slower read of the same bytes. A cache hit and a cache miss return identical results; the only difference is latency.

Here there are two distinct caches doing two different jobs:

Recent conversation context (Redis). This is a conventional cache — the turns exist in the database too, and Redis just avoids the round trip. Standard.

Generated responses. This one is not conventional at all. The response does not exist anywhere else; it was produced by burning GPU time. Caching it is not avoiding a read, it is avoiding a computation that costs roughly a million times more than the read would.

That inverts the economics of caching completely. In the distributed cache chapter, a miss cost you a few milliseconds of database latency. Here a miss costs seconds of GPU occupancy on hardware in the ten-thousand-unit fleet from the last lesson. Which means:

A far lower hit rate is still worth having. Even 5% is enormous when each hit saves a GPU-second.
Approximate matching becomes worthwhile — which is exactly why the next lesson reaches for a semantic cache rather than an exact-key one.
Correctness gets harder, because a hit and a miss no longer return the same thing.

When the cached item is expensive to produce rather than merely slow to fetch, every caching rule you learned changes. That is the single most useful transfer from the classic building blocks to this domain.

The CDN is here for what the model does not produce

Easy to skim past, but the detailed design places it deliberately: the CDN serves UI assets — scripts, styles, fonts, images — while "generated responses from the model are handled separately."

That separation is the point. The CDN cannot cache a response, because responses are unique per user and streamed as they are produced. Its entire job is keeping everything that is not generation off the expensive path.

Compare the YouTube chapter, where the CDN carried the product itself — the video bytes — and was the dominant infrastructure. Here it carries the wrapper around the product and is almost incidental.

That contrast is a good summary of the whole chapter. The valuable payload has moved from bytes that can be replicated to computation that cannot.

"Graph-structured data" is a copy-paste artifact — but the database question is real

The database line says "store metadata and graph-structured data." There is no graph anywhere in this design — that phrasing has drifted in from a chapter where there was one. Take the intent as "metadata."

The real storage question is more interesting anyway, because this design ends up with four stores of different shapes, each chosen for the shape of what it holds:

Store	Holds	Why this shape
MongoDB	Session metadata, billing history	Flexible schema, horizontally scalable
Vector DB	Embeddings of history and knowledge	Similarity search, not key lookup
Redis	Recent conversation turns	Sub-millisecond, on the critical path
Relational	Structured feedback — ratings, thumbs	Fixed schema, aggregate queries

The vector database is the genuinely new one. Every store in this course so far retrieved by key or range. A vector database retrieves by nearest neighbour in an embedding space — it answers "what is most similar to this?" rather than "what is at this key?"

That is not a variation on existing indexes; it is a different query model, and it is what makes both semantic caching and retrieval possible. It arrives in the next two lessons.

ZooKeeper coordinating model servers has a twist worth naming

ZooKeeper appears where you would expect — "tracking active instances, distributing workloads, and handling failovers." Same role as the primary-replica mapping in the Quora chapter or segment assignment in Google Maps.

The twist is how slow recovery is. Every earlier chapter could treat replacing a failed node as fast: start a process, let it warm up, resume serving.

Here, bringing a model server into service means loading 140 GB of weights into GPU memory. That is minutes. So the coordination layer cannot rely on reactive replacement — it needs warm standby capacity already holding the weights, which is expensive precisely because that hardware is the expensive kind.

It also means failure is costly for in-flight work. A web request lost to a crash is retried in milliseconds. A sequence that has generated 800 tokens and loses its server has lost 800 forward passes of accumulated KV cache, and must start over.

Key takeaway

Five blocks — load balancers, databases, CDNs, pub-sub, cache — plus an API gateway and ZooKeeper. Four behave as before. The cache does not: it stores output that was computed, not fetched, so a miss costs GPU-seconds rather than milliseconds and every caching rule shifts — lower hit rates pay off, and approximate matching becomes worthwhile. The CDN carries only the wrapper, since responses cannot be cached at the edge. And ZooKeeper's recovery is minutes, not seconds, because a replacement server must load 140 GB of weights.

Next: the high-level design and the path a prompt takes through it.

Building Blocks We Will Use

Enjoying the preview?