High-Level Design

Why this matters: the high-level flow is short enough to state in five steps, which makes it a good place to notice what is not in it.

The high-level design illustrates how the system handles real-time conversations.

Step	What happens
1. User input	The user submits a text prompt via the interface or API
2. Gateway processing	The API gateway authenticates the request, applies rate limiting, manages the session, and forwards the prompt to the model server
3. Model inference	The AI model processes the prompt using conversation history. Responses are cached for retrieval and logged in the database
4. Response delivery	The generated response is returned to the user via the API gateway
5. Feedback loop	User feedback is collected to improve system performance and fine-tune future models

The gateway does four jobs, and rate limiting is the load-bearing one

Authentication, rate limiting, session management, and routing. In most designs rate limiting is a defensive measure against abuse. Here it is cost control, and it is the difference between a business and a bankruptcy.

Lesson 4 established that a single response can occupy expensive hardware for seconds. So an unlimited-throughput client is not merely rude — it is a direct, uncapped charge against the most expensive resource the company owns. A user hitting the API in a tight loop consumes GPU time far faster than any conventional endpoint could consume anything.

That changes what the limit should be measured in. Requests per minute is the wrong unit, because requests are not equal — one might generate 20 tokens and another 2,000, a hundredfold difference in cost.

Real systems therefore limit on tokens rather than requests, usually on both dimensions at once: tokens per minute and requests per minute, per user and per organization. Which is also, not coincidentally, the unit that billing uses.

This is a genuinely distinctive point about AI systems. The rate limiter, the billing system, and the capacity planner all measure the same quantity — tokens — because tokens are simultaneously the unit of work, the unit of cost, and the unit of price.

Step 5 runs on a different clock from steps 1 through 4

Steps 1 to 4 complete in seconds. Step 5 does not close for weeks.

Feedback improves the system through fine-tuning or retraining, which is an offline batch process. The user who pressed thumbs-down gets no benefit from having done so — their response is already delivered and unchanged.

So the diagram's neat loop conceals two systems with almost nothing in common:

	Serving path	Improvement loop
Latency	Seconds	Weeks
Trigger	User request	Scheduled retraining
Failure impact	User sees an error	Nobody notices for a while
Scaling unit	GPUs for inference	GPUs for training
Consistency	Must be immediate	Eventual by many orders of magnitude

Drawing them as one loop is the most misleading thing about this diagram. In practice the feedback path is fully asynchronous — which is exactly why the detailed design puts pub-sub in front of it.

The generalizable point: when a diagram shows a cycle, check whether every arrow runs at the same speed. If one arrow is a million times slower than the others, it is not part of the same system, and treating it as though it were leads to designing synchronous machinery for something that should be a queue and a batch job.

What the high-level design omits

Compare this diagram with the detailed one two lessons ahead and note what is missing here, because each omission is a decision the detailed design has to make:

No load balancer — the gateway appears to talk to "the AI model" as a single thing, when it is thousands of GPUs needing careful placement.
No content moderation — a whole component sits between generation and delivery in the detailed design, adding latency to every response.
No vector database — so retrieval, which is how "using conversation history" actually works, is invisible.
No streaming — step 4 shows a response being "returned," implying a single payload, when Lesson 4 showed the response takes 25 seconds and must be streamed.

That last one is the most consequential. A high-level design that shows request-then-response has quietly assumed the response is atomic. It is not, and once you accept that tokens flow continuously, the connection model, the moderation placement, and the failure semantics all change.

None of this makes the diagram wrong — high-level designs elide. But knowing what a diagram elides is what lets you talk about it rather than recite it.

Where the model gets its context

Step 3 says the model "processes the prompt using conversation history." That phrase is doing an enormous amount of work.

Assembling the prompt is the real step 3

Remember from Lesson 1 that the model is stateless. So before any inference happens, something must build the actual input, and it is assembled from several sources:

[system instructions]        <- fixed policy and persona
[user profile / preferences] <- from the user profile service
[recent conversation turns]  <- from Redis
[retrieved relevant context] <- from the vector database
[the current prompt]         <- from the user

This assembly step is where personalization, dialogue management, and NLU from the requirements list actually happen. It is invisible in the diagram and it is most of the interesting engineering.

It also carries a cost the diagram cannot show. Everything prepended is prefill work — tokens the model must process before generating anything. Doubling the context you include roughly doubles time-to-first-token and adds KV cache that displaces other users' concurrency.

Which sets up the central trade of the retrieval lesson: more context makes better answers and costs more on every single request. The engineering is deciding what to leave out.

Key takeaway

Five steps: prompt, gateway, inference, delivery, feedback. The gateway's rate limiting is cost control, and it should measure tokens rather than requests — the same unit as billing and capacity planning. Step 5 belongs to a different system running weeks behind the other four, which is why it is asynchronous. And the diagram elides the load balancer, moderation, retrieval, and — most importantly — streaming, since a 1,250-token response is not an atomic payload. The real step 3 is assembling the prompt from system instructions, profile, recent turns, and retrieved context, every token of which costs prefill time.

Next: why an ordinary cache fails here, and what replaces it.

High-Level Design

Where the model gets its context

Enjoying the preview?