Detailed Design: The Components

In one line: the component list is where the design becomes specific. Read it for which components sit on the critical path, because that determines the latency budget and, in this system, the cost.

Requests are routed through the API gateway to a load balancer, which distributes traffic across application servers.

The components

Sort them by whether they sit on the critical path

This is the most useful thing to do with the list, because in this system critical-path components spend the expensive resource and off-path ones do not.

Everything in the left column adds to time to first token. Everything in the right column is free, latency-wise, because it happens after or beside the response.

Two observations follow.

The right column exists because of a deliberate decision. Billing, logging, and feedback could all have been synchronous, and in a naive design they would be. Putting them behind pub-sub is what the design means by "ensure events propagate without blocking API calls." Notice especially that billing is asynchronous — the system delivers the response first and charges afterwards, accepting a small revenue risk to protect latency.

The left column has one entry that should give you pause. Content moderation sits between generation and delivery, which means it adds latency to every single response — and, as the next lesson shows, it interacts badly with streaming. That is the subject of the chapter's sharpest design question.

The components worth interrogating

The model server is not one thing

"The core processing unit hosting AI models" is doing a lot of hiding. From Lesson 4, a model server is:

• A shard of a model — a 175B model in FP16 needs 350 GB, so it spans several GPUs, and "a model server" is really a coordinated group.
• A scheduler running continuous batching, admitting and evicting sequences every step.
• A memory allocator managing paged KV cache, which is the actual limit on concurrency.
• A streaming endpoint holding a connection open for the seconds generation takes.

Note the plural in the diagram — "model servers" — and that ZooKeeper coordinates them. That is right, but the coordination problem is harder than the usual one: these servers are not interchangeable. They may host different models, different quantizations, or different shards of one model. Routing must know which server can serve which request, not merely which is least loaded.

And a strong design would host more than one model. A small fast model for simple prompts, a large one for hard prompts, and a classifier deciding between them. That single decision can cut the fleet from Lesson 4 substantially, because most prompts are easy.

The vector database does two different jobs and the design conflates them

It stores "vector embeddings of conversation history and external knowledge." Those are not the same feature, and separating them is worth doing:

Conversation history — the user's own past turns. This is memory: recalling something the user said last week without holding it all in context. Per-user, private, written on every turn.

External knowledge — documents, manuals, articles the model was not trained on. This is retrieval-augmented generation: grounding answers in a corpus. Shared across users, read-only at request time, updated by a separate ingestion pipeline.

They differ in almost every operational dimension — privacy, write rate, sharding key, update path. A production system usually keeps them as separate indexes even when they share a database engine.

Conflating them causes a real bug worth naming: if one index serves both, a retrieval for "external knowledge" can surface another user's conversation. Per-user isolation is not optional, and it is the kind of thing that is easy to get wrong and catastrophic when you do.

Redis for conversation history is the one genuinely conventional cache here

After Lesson 7's semantic cache, this one is refreshingly ordinary: "temporarily stores recent conversation history to prevent redundant database lookups."

Recent turns live in Redis; older ones in durable storage. A hit and a miss return the same data, so all the normal caching intuitions apply.

It works well because of the access pattern Lesson 3 identified: ten requests per user per day arriving in bursts within a conversation. Recency is an excellent predictor here, which is exactly the condition under which LRU shines.

The subtlety is what "recent" should mean. Conversations have no natural end — a user may return after a week and expect continuity. So eviction is a genuine product decision, not just a memory one: evict too eagerly and the system forgets mid-conversation, keep too much and you pay for context nobody will use. The usual resolution is a tiered one — Redis for the active session, durable storage for the transcript, vector database for semantic recall of anything older.

MongoDB's justification is thinner than it looks

"Selected for its flexible data models and horizontal scalability."

That is a generic argument, and it sits oddly next to what MongoDB is holding here: session metadata and billing history. Billing records are exactly the kind of data with a fixed schema and transactional requirements — you want atomicity when charging a customer, and the workflow section confirms it stores "billing history."

Note that the design already uses a relational database for structured feedback, so the reflex to reach for one exists in this design; it just was not applied to billing.

The defensible split is by shape rather than by preference:

• Billing and transactions — relational, for atomicity and auditability.
• Session metadata and conversation transcripts — document store, since turn structure varies and schema evolves.

Storage volume is not the deciding factor either way. Lesson 3 established the whole system stores about 1.1 PB per year, which is small enough that either engine handles it comfortably. When scale is not the constraint, choose the store for its consistency guarantees rather than its scaling story — and billing is the clearest case in this design for wanting transactions.

ZooKeeper's job here is heavier than the usual mapping table

"Tracking active instances, distributing workloads, and handling failovers" is familiar — that building block used ZooKeeper for primary-replica mappings, Maps used a key-value store for segment-to-server.

Two things make it harder here.

Recovery is minutes, not seconds. Replacing a model server means loading tens or hundreds of GB of weights into GPU memory. So failover cannot be reactive in the usual sense — you need warm standby capacity already holding the weights, and that standby is the most expensive hardware you own.

In-flight work is expensive to lose. A dropped web request is retried in milliseconds. A sequence 800 tokens into generation has 800 forward passes of accumulated KV cache on that machine, and losing it means starting over — with the user having already read 800 tokens they will now see contradicted.

That second point argues for something the design does not mention: draining rather than killing. When taking a model server out of rotation for a deploy, stop admitting new sequences and let existing ones finish. Restarts should be measured in conversations, not processes.

Next: the full journey of a prompt through these components.