What Sits Around the Model

In one line: the engine that runs the model is largely a solved, off-the-shelf component, and everything interesting in an inference platform is the machinery you put in front of it.

The engine is one box

Modern serving engines already handle continuous batching, paged KV cache, tensor parallelism and quantised weights. You configure them rather than build them, and a candidate who spends twenty minutes on attention kernels is answering a question nobody asked.

What you are being asked to design is the layer that decides which model handles a request, whether it should run at all, where it runs, and what happens when it cannot.

Every box exists because of a specific pressure, and being able to name the pressure rather than the component is the difference between drawing a diagram and designing one.

Why this layer matters more than usual

Three properties make LLM serving different from serving a normal stateless API, and they justify the whole platform.

**Capacity is fixed on a short horizon. You cannot add GPUs in ninety seconds. Accelerators are scarce, expensive and often reserved months ahead, so the platform's job is to make the fixed capacity you have behave well — not to scale out of a problem.

Requests are wildly non-uniform. Two calls to the same endpoint can differ in cost a thousandfold. A load balancer treating them as interchangeable will make bad decisions, and a rate limit counting requests is measuring the wrong thing.

Every request holds state.** The KV cache means an in-flight request occupies GPU memory that grows with its conversation. That single fact breaks stateless load balancing, which is the subject of its own lesson.

That closing box is the chapter in one diagram. Each of the three familiar techniques it names gets a lesson explaining precisely why it fails and what replaces it.

The two paths through the system

Worth separating early, because they have different requirements and candidates often design only one.

**Interactive. A user is waiting. Time to first token dominates, streaming is mandatory, and the right failure mode is a fast degradation rather than a slow correct answer.

Batch. No one is waiting. Throughput and cost per token are everything, latency is irrelevant, and the work can be scheduled into whatever capacity is idle.

The design consequence is that these two should share hardware and not share a queue. Batch work is what fills the trough that interactive traffic leaves empty — which, as the estimation chapter showed, is what moves average utilisation and therefore the entire cost picture.

What good looks like

A serving platform is doing its job when four things are true, and they make a reasonable checklist to design against:

Cheap requests are cheap. A trivial question does not consume frontier-model capacity.

Repeated work is not repeated. Identical and near-identical requests, and shared prompt prefixes, are not recomputed.

Overload degrades some requests, not all of them. At capacity the system sheds or queues by priority rather than slowing everyone equally.

A single failure does not take the product down.** There is a path that still works when the primary model or the primary region is unavailable.

Next: the latency contract that streaming creates.