What Makes an LLM System Different

In one line: every design so far has shared a hidden assumption — that serving a request is cheap, and the hard part is having the right data in the right place. That assumption is false here, and almost everything follows from its failure.

What the system is

ChatGPT is an advanced conversational application built on a sophisticated AI language model developed by OpenAI. It understands natural language and generates human-like text to assist with various communication tasks.

It helps users do three things: simplifying technical subjects, proposing innovative ideas, and composing clear, well-structured text.

The behaviour worth noticing is in the follow-up. A conversation looks like this:

User:  Who wrote Harry Potter?
Model: J.K. Rowling
User:  What other books has she written?
Model: She also wrote The Casual Vacancy and the Cormoran Strike series.

ChatGPT-like systems are notable for interpreting incomplete or ambiguous inputs to provide relevant, context-aware responses. Instead of relying on predefined answers, these systems use generative AI and large language models to analyze intent and patterns.

"She" is the entire system requirement in one word

The second question — "What other books has she written?" — is meaningless in isolation. There is no "she" in it. Answering requires the previous turn.

This is the smallest possible illustration of the design's central obligation: the model itself is stateless. Each call is a pure function from input to output. It has no memory of the previous exchange.

So the conversational continuity users experience is not a model property — it is an architectural one. Something in the system must retrieve the prior turns and prepend them to the prompt on every single call.

That single fact generates three components you will meet later: a cache for recent turns, a vector database for older or semantically related ones, and a user profile service for durable preferences. All three exist to reconstruct, on every request, the context the model cannot hold itself.

Hold onto the consequence: context is not free. Every prior turn you prepend is input the model must process again, and in the next lesson but one we will see exactly what that costs.

What we are actually designing

In this chapter, we will design a text-to-text generation system. While OpenAI's specific architecture is proprietary, we will apply general principles and best practices for building similar systems.

Scoping out the model is the right move — and worth saying explicitly

We are designing the system around a language model, not the model. Training, architecture, and weights are given.

That is the same scoping decision that building block made when it assumed "road data comes from external sources." In both cases an enormous, genuinely separate problem is bounded off so the serving problem can be designed properly.

Say this out loud in an interview. A candidate who begins describing transformer attention when asked to design ChatGPT has misread the question — the interviewer wants the serving system. But a candidate who never mentions the boundary sounds unaware that it exists.

The boundary is not arbitrary, either. It is exactly where the iteration loops differ: model training is a weeks-long offline loop, serving is a millisecond-scale online one, and the two are connected only by a deployment step and a feedback dataset.

The three assumptions this chapter breaks

Every previous design in this course rested on assumptions that no longer hold.

Assumption	Everywhere else	Here
Serving a request is cheap	A web server handles ~64,000 RPS. Compute is nearly free; the cost is I/O	A single response can occupy a GPU worth tens of thousands of dollars for seconds
A request is one unit of work	Request arrives, work happens, response returns. Latency is a single number	A response is generated one token at a time, each token a separate forward pass
Correct output is well-defined	The route is shortest or it is not. The video plays or it does not	Output is plausible-sounding text that may be confidently wrong, and no component can tell

The first two assumptions invert the entire cost model

Take them in turn, because they compound.

Serving is expensive. In that building block the interesting number was 12 Tb/s of egress; compute barely featured. Here, as the next lesson shows, the bandwidth is under 1 Gb/s total — a rounding error — while the compute bill is the entire business. The scarce resource has moved.

A response is not one operation. Language models generate autoregressively: to produce token 50, the model must have already produced tokens 1 through 49, because each one is fed back in as input. A 1,000-token answer is 1,000 sequential forward passes through a network with billions of parameters.

That sequential dependency is the hard constraint of the whole chapter. It means:

You cannot parallelize a single response. More GPUs make a response possible, not faster to the last token.
Latency has two numbers, not one — time to the first token and time between subsequent tokens.
A response occupies its GPU for the whole duration, unlike a web request that occupies a thread for milliseconds.

Compare it to that building block, where Dijkstra's outward exploration could not be parallelized without splitting the graph. Same shape of problem: an inherently sequential algorithm meets a latency budget. The resolutions differ — Maps precomputed, and here we will batch — but recognizing the shape is what lets you reach for the right tool.

The third assumption is the one with no engineering answer

Every prior chapter had a definition of correct. A shortest path is shortest. A blob returns the bytes that were written. Even the ETA in that building block, which we called observational, could be checked against the actual arrival time.

Here, the model can produce fluent, confident, well-formatted text that is simply false, and no component in the architecture can detect it. The content moderation system checks for policy violations, not truth. The cache cannot know a response is wrong. The load balancer certainly cannot.

This matters for the design in a specific way: correctness is not in the requirements list at all. The next lesson enumerates four functional and five non-functional requirements, and factual accuracy is not among them. That is a real omission and one of the strongest things you can raise in an interview.

The partial answers — grounding responses in retrieved documents, citing sources, evaluation harnesses that score outputs offline, human review of sampled traffic — are all mitigations, not solutions. Naming them as mitigations is more credible than claiming any of them fixes the problem.

The organizing question

Here is the thread to follow through the chapter:

Given that generating a response is expensive and inherently sequential, how does the system avoid doing it?

Nearly every component answers some version of that:

Semantic caching — return a previous response instead of generating a new one.
Redis for recent context — avoid re-fetching history from durable storage.
Batching — amortize one expensive forward pass across many concurrent users.
Retrieval — supply a few relevant facts instead of a vast conversation history.
CDN for static assets — keep everything that is not generation off the expensive path.

Key takeaway

An LLM system breaks three assumptions the rest of the course relied on: serving is expensive rather than nearly free, a response is many sequential forward passes rather than one operation, and correctness is undefined — output can be fluent and wrong with no component able to tell. The model is stateless, so every appearance of conversational memory is architecture. And the thread through every component is the same question: how do we avoid generating?

Interview signal by level

Level	What a strong answer sounds like
L4	"It's a conversational system — the user sends a prompt, a model generates a response, and we keep conversation history for context."
L5	Names the statelessness: "the model has no memory, so context is an architectural responsibility — we retrieve prior turns and prepend them on every call, which means context costs compute."
Staff+	Leads with the inverted cost model: "this is the first design here where compute is the scarce resource, not storage or bandwidth. And generation is autoregressive — a 1,000-token answer is 1,000 sequential forward passes, so a single response can't be parallelized and latency splits into time-to-first-token and inter-token time. That sequential constraint drives batching, caching, and retrieval — every one of them is a way to avoid generating. I'd also flag up front that factual correctness isn't in the requirements and has no architectural fix, only mitigations."

Next: the requirements, and what is conspicuously absent from them.