The Request Workflow

In one line: the workflow reads as a clean sequence. Tracing where the time goes turns it into a design you can reason about.

The journey

When a user enters a prompt through a web or mobile interface, the request first reaches the API gateway, which manages authentication, authorization, and rate limiting. From there, a load balancer distributes the request across multiple application servers.

Before reaching the model, the prompt passes through the preprocessing NLU service. If the user has interacted before, a user profile service retrieves relevant historical data to personalize the response. Once preprocessed, the request is sent to the model servers, where the large language model generates a response. When additional context is required, the system queries the vector database to retrieve semantically relevant information from past interactions.

Before the response is returned, it passes through the content moderation system to filter out harmful or inappropriate content. ZooKeeper coordinates the model servers by managing configuration, leader election, and health monitoring during scaling and failover.

If the request involves a paid feature, the payment processing system logs the transaction and updates records in MongoDB. Meanwhile, a CDN ensures static assets load quickly, but generated responses from the model are handled separately.

Finally, the response is sent back to the user. If the user provides feedback, it is collected via the feedback module and stored in the feedback database. Structured feedback (ratings, thumbs up/down) is stored in a relational database. Unstructured feedback (comments, session metadata) is stored in a document database.

Where the time actually goes

Trace the latency budget and the shape of the system becomes obvious

Attach rough costs to each step. The exact figures depend on the deployment; the ratios do not.

Step	Rough cost	Share
Gateway — auth, rate limit	~1 ms	negligible
Load balancer	under 1 ms	negligible
NLU pre-processing — embed	~10 ms	negligible
Profile lookup	~5 ms	negligible
Redis — recent turns	~1 ms	negligible
Vector DB — ANN retrieval	~20 ms	negligible
Model prefill	~200 ms	first token
Model decode, 1,250 tokens	~25 s	dominant
Moderation	~50 ms	see below

Everything before the model — the gateway, the load balancer, the NLU service, the profile lookup, Redis, and retrieval — costs perhaps 40 milliseconds combined. Generation costs 25 seconds. The model is roughly 600 times everything else put together.

Two consequences that should shape how you talk about this design:

Optimizing the non-model path is nearly pointless. Shaving 10 ms off retrieval improves total latency by 0.04%. In every earlier chapter, this is where the engineering went; here it is noise.

But it is not free — because retrieval does not just cost its own 20 ms, it adds tokens to the prompt, which adds prefill time and consumes KV cache. The cost of a component here is not its own latency but the work it creates for the model. That reframing is the single most transferable idea in the lesson.

What the linear narrative hides: several steps are concurrent

The prose reads as a chain, but the fast steps do not depend on each other:

Profile lookup, Redis, and embedding can all run in parallel. Only retrieval is genuinely dependent, since it needs the embedding first.

They then converge on the step the narrative never names: assembling the prompt. That is where personalization and dialogue management actually happen, and it is the decision point for how many tokens the model will have to process.

It is worth saying explicitly in an interview, because the design's own phrasing — "the model processes the prompt using conversation history" — makes it sound automatic. It is not. Something chose what to include, and that choice is the main lever on both quality and cost.

Streaming changes the shape of everything after generation

"Finally, the response is sent back to the user" cannot be literally true

At 1,250 tokens, a complete response takes roughly 25 seconds to generate. No user waits 25 seconds for a blank screen, so tokens are streamed as produced — the user starts reading at about a second.

The workflow describes a request-response shape, and reality is a stream. That has consequences the narrative does not address:

Moderation cannot simply "run before the response is returned." By the time the last token exists, the first thousand have already been read. The next lesson takes this on directly.

The connection is held for the whole generation. From Lesson 3, the byte volume is trivial — under 1 Gb/s — but tens of thousands of connections stay open for tens of seconds. That is the C10K shape again, met in the YouTube and Maps chapters for different reasons, and it argues for event-driven servers over thread-per-connection.

Failure mid-stream is a new failure mode. If a model server dies 800 tokens in, the user has already read 800 tokens. You cannot silently retry, because a retry produces different text — generation samples. The honest options are to stop with an error, or to continue with the partial output as context, and both are visible to the user. There is no clean recovery.

Cancellation matters financially. A user who closes the tab at token 100 of 1,250 has stopped caring, but the GPU keeps working unless something tells it to stop. Propagating cancellation back to the scheduler frees the slot immediately, and at ten thousand GPUs that is not a small saving.

After the response

Billing happens after delivery, and that is a deliberate trade

"If the request involves a paid feature, the payment processing system logs the transaction." Via pub-sub, so asynchronously.

The reasoning is sound, and it is specific to this domain: you cannot know the cost until the response is finished, because pricing is per token and the token count is not known in advance. So synchronous charging is not merely slow, it is impossible without generating first.

The trade is that a user can consume compute the system then fails to bill for, if the event is lost between generation and the billing service. That is a real revenue leak, mitigated by durable pub-sub and reconciliation rather than eliminated.

The stronger pattern in production combines the two: check a quota or credit balance before generating — cheap, synchronous, prevents runaway spend — then meter the exact cost afterwards. Pre-authorization plus post-settlement, which is exactly how card payments work, and for the same reason: the final amount is not known at authorization time.

The feedback split by shape is a small, correct decision

Structured feedback (ratings, thumbs up/down) goes to a relational database. Unstructured feedback (comments, session metadata) goes to a document database.

This is the right call and worth a sentence on why. Thumbs up/down is a fixed schema queried in aggregate — "what fraction of responses on topic X were rated positively this week" — which is precisely what relational engines and columnar aggregation are good at.

Free-text comments have no fixed shape and are read individually or fed to analysis, which suits a document store.

It also quietly notes the design has four data stores now — MongoDB, vector DB, Redis, relational — each justified by the shape of what it holds. That is the correct instinct, though as Lesson 8 argued, it makes MongoDB holding billing harder to defend when a relational store is already in the architecture.

Key takeaway

The model is roughly 600 times the cost of everything else combined, so optimizing the non-model path is nearly pointless — but components are not free either, because their real cost is the tokens they add to the prompt. Several fast steps run in parallel, converging on the unnamed but decisive step of assembling the prompt. Because responses stream, "finally the response is sent" is false: moderation cannot simply run before delivery, connections stay open for tens of seconds, mid-stream failure has no clean retry since generation samples, and cancellation must reach the scheduler to free the GPU. Billing is necessarily post-hoc because token count is unknown in advance, which argues for pre-authorization plus post-settlement.

Next: the moderation question, which is the sharpest in the chapter.