High-Level Design

In one line: the high-level flow is short enough to state in five steps, which makes it a good place to notice what is not in it.

The high-level design illustrates how the system handles real-time conversations.

The gateway does four jobs, and rate limiting is the load-bearing one

Authentication, rate limiting, session management, and routing. In most designs rate limiting is a defensive measure against abuse. Here it is cost control, and it is the difference between a business and a bankruptcy.

Lesson 4 established that a single response can occupy expensive hardware for seconds. So an unlimited-throughput client is not merely rude — it is a direct, uncapped charge against the most expensive resource the company owns. A user hitting the API in a tight loop consumes GPU time far faster than any conventional endpoint could consume anything.

That changes what the limit should be measured in. Requests per minute is the wrong unit, because requests are not equal — one might generate 20 tokens and another 2,000, a hundredfold difference in cost.

Real systems therefore limit on tokens rather than requests, usually on both dimensions at once: tokens per minute and requests per minute, per user and per organization. Which is also, not coincidentally, the unit that billing uses.

This is a genuinely distinctive point about AI systems. The rate limiter, the billing system, and the capacity planner all measure the same quantity — tokens — because tokens are simultaneously the unit of work, the unit of cost, and the unit of price.

Step 5 runs on a different clock from steps 1 through 4

Steps 1 to 4 complete in seconds. Step 5 does not close for weeks.

Feedback improves the system through fine-tuning or retraining, which is an offline batch process. The user who pressed thumbs-down gets no benefit from having done so — their response is already delivered and unchanged.

So the diagram's neat loop conceals two systems with almost nothing in common:

Drawing them as one loop is the most misleading thing about this diagram. In practice the feedback path is fully asynchronous — which is exactly why the detailed design puts pub-sub in front of it.

The generalizable point: when a diagram shows a cycle, check whether every arrow runs at the same speed. If one arrow is a million times slower than the others, it is not part of the same system, and treating it as though it were leads to designing synchronous machinery for something that should be a queue and a batch job.

What the high-level design omits

Compare this diagram with the detailed one two lessons ahead and note what is missing here, because each omission is a decision the detailed design has to make:

• No load balancer — the gateway appears to talk to "the AI model" as a single thing, when it is thousands of GPUs needing careful placement.
• No content moderation — a whole component sits between generation and delivery in the detailed design, adding latency to every response.
• No vector database — so retrieval, which is how "using conversation history" actually works, is invisible.
• No streaming — step 4 shows a response being "returned," implying a single payload, when Lesson 4 showed the response takes 25 seconds and must be streamed.

That last one is the most consequential. A high-level design that shows request-then-response has quietly assumed the response is atomic. It is not, and once you accept that tokens flow continuously, the connection model, the moderation placement, and the failure semantics all change.

None of this makes the diagram wrong — high-level designs elide. But knowing what a diagram elides is what lets you talk about it rather than recite it.

Where the model gets its context

Step 3 says the model "processes the prompt using conversation history." That phrase is doing an enormous amount of work.

Assembling the prompt is the real step 3

Remember from Lesson 1 that the model is stateless. So before any inference happens, something must build the actual input, and it is assembled from several sources:

[system instructions]        <- fixed policy and persona
[user profile / preferences] <- from the user profile service
[recent conversation turns]  <- from Redis
[retrieved relevant context] <- from the vector database
[the current prompt]         <- from the user

This assembly step is where personalization, dialogue management, and NLU from the requirements list actually happen. It is invisible in the diagram and it is most of the interesting engineering.

It also carries a cost the diagram cannot show. Everything prepended is prefill work — tokens the model must process before generating anything. Doubling the context you include roughly doubles time-to-first-token and adds KV cache that displaces other users' concurrency.

Which sets up the central trade of the retrieval lesson: more context makes better answers and costs more on every single request. The engineering is deciding what to leave out.

Next: why an ordinary cache fails here, and what replaces it.