The Inference Path
In one line: the chapter names the right serving techniques — vLLM, batching, KV cache, quantization. What it does not do is trace what they imply for routing, and that implication contradicts a stated design choice.
The path
The request reaches the GPU inference cluster, where a model serving framework such as vLLM performs batched inference. By grouping multiple requests into a single GPU execution and leveraging KV cache optimization, the system improves throughput while keeping latency low.
As soon as generation begins, tokens are streamed back to the IDE using SSE. This reduces time to first token, even if full response generation takes longer.
Naming vLLM is naming continuous batching, and that is the right answer
The chapter says "a model serving framework such as vLLM performs batched inference" without saying which kind, and the distinction is the whole value.
STATIC batching: wait to accumulate N requests -> run to completion
-> a request arriving just after a batch departs WAITS
-> every request finishes when the LONGEST one does
CONTINUOUS batching: maintain a running batch; requests JOIN and LEAVE
at token boundaries
-> no waiting to start, no waiting for the slowest peer
Static batching is incoherent under a 300 ms budget — waiting to fill a batch can consume the entire budget before inference begins.
Continuous batching works because decode proceeds token by token, so the server can admit a new request at any step and evict a finished one immediately. That is what makes the GPU tier's throughput figures achievable without paying a queueing penalty, and it is why "vLLM" is a meaningful answer rather than a brand name.
Continuous batching is what lets a latency-critical service use batching at all — and it is the reason Lesson 3's budget shows only 0–20 ms of queue wait.
KV cache is mentioned as a throughput feature, and it is the mechanism the budget depends on
"...leveraging KV cache optimization, the system improves throughput while keeping latency low."
That sentence treats it as one optimization among several. Lesson 3 showed it is the optimization:
Without prefix reuse: 4,000-token prefill -> ~1,300 ms -> budget blown With prefix reuse: ~20-token delta -> ~10 ms -> budget closes
Two orders of magnitude, on the stage that dominates time-to-first-token.
And it has a consequence the design does not draw. The prefix's computed attention state lives in the memory of one specific GPU node. For the next request from that developer to reuse it:
The request must arrive at THE SAME NODE.
Which means session affinity in the load balancer — and the design says the opposite:
"Load balancers distribute traffic across GPU clusters deployed in multiple geographic regions."
Undirected distribution and prefix reuse are incompatible. A request round-robined to a different node finds a cold cache and pays the full prefill.
The resolution is routing on a session or file key rather than round-robin:
route_key = hash(session_id) -> consistent hashing across GPU nodes
-> same developer, same node, warm prefix
-> rebalance on node loss (and accept a cold prefill)
This is the same shape as that building block's per-document queues and that building block's partition-by-entity: when state that makes a request fast lives on one node, routing must find that node.
A stateless-looking inference tier that depends on KV cache reuse is not stateless — and pretending otherwise silently discards the optimization that makes the budget achievable.
KV cache competes with batch size for the same GPU memory
The trade the design never mentions, and it is a real capacity decision.
GPU memory holds three things: model weights, KV cache for in-flight requests, and KV cache retained for warm prefixes. The first is fixed; the other two compete.
More retained prefixes -> more warm sessions -> better TTFT
-> LESS memory for batching -> lower throughput
Larger batches -> better GPU utilization, lower cost per request
-> fewer retained prefixes -> more cold prefills
So "how many sessions do we keep warm?" is the same question as "how large can our batches be?", and both are answered by the same fixed pool of memory.
This is also where quantization earns its place beyond cost. The design mentions INT8 for "reduced memory usage and improved throughput" — and the more valuable consequence is that halving the weights frees memory for both batching and prefix retention.
FP16 weights: less headroom -> smaller batches AND fewer warm sessions INT8 weights: more headroom -> larger batches AND more warm sessions
Quantization buys latency indirectly, by freeing the memory that the latency optimizations consume — which is a stronger argument for it than the cost saving the chapter cites.
The corresponding eviction question is a latency question too: evicting a prefix converts a 10 ms request into a 1,300 ms one, so eviction policy is a tail-latency policy. Least-recently-used by session is the sensible default, since a developer who stopped typing five minutes ago is the cheapest to evict.
Graceful degradation
If inference fails or exceeds latency thresholds, the system returns no suggestion instead of an error. Cached responses provide a partial fallback for frequently seen requests.
Returning nothing is the correct failure mode, and it is worth saying why explicitly
This is the design's best reliability decision and the reasoning generalizes.
An error in an editor is worse than silence, because an error demands attention. A toast notification, a red squiggle, or a modal interrupts the developer — and the entire product premise is not interrupting. Silence costs nothing: the developer keeps typing and never knew a request was made.
Error shown: the developer stops, reads, dismisses -> the failure INTERRUPTED Nothing shown: the developer types on -> the failure was INVISIBLE
The chapter's quiz makes the same point by elimination, and the wrong answers are instructive:
Retrying with backoff is incoherent here. By the time a retry lands, the context is stale — you would be answering a file that no longer exists. Retries are for operations whose input is still valid, and under a 300 ms budget against moving input, it never is.
Queuing for later delivery is worse: a suggestion arriving after the developer has moved on is not late, it is wrong, and displaying it is an interruption with no value.
A latency budget set by changing input makes retry and queue-for-later actively harmful, not merely useless — the same conclusion the typeahead chapter reached.
One refinement: the timeout must be client-side as well as server-side. If the server is slow, the plugin should abandon the request at its own deadline rather than waiting, and cancel it so the GPU stops working on an answer nobody will see.
What is missing from the inference path
Three things, in descending order of value.
Speculative decoding (Lesson 3) — a small draft model proposes tokens that the large model verifies in one pass. Code is its best case, because closing brackets, boilerplate, and repeated identifiers are highly predictable. It reduces decode time, which matters most for explainCode.
Request cancellation. When the developer types again, the in-flight request is answering a stale prefix. Cancelling frees GPU capacity immediately and is free capacity at exactly the moment the system is busiest — the design mentions debouncing but not cancellation.
Model tiering. that building block had a cost-aware router; this design has one model. A short single-line completion and a full function body have very different requirements, and a small fast model handles the first at a fraction of the cost:
Single-line, high-confidence completions -> small model, ~50 ms Multi-line function generation -> full model explainCode -> full model, generous context
The tiering opportunity is larger here than in the support bot, because inline completions are both the most frequent request and the most latency-critical — exactly the class that most benefits from a cheap fast path.
| Technique | Named in the design? | What it actually does here |
|---|---|---|
| Streaming (SSE) | ✅ Central | Makes the budget about TTFT rather than completion |
| Continuous batching | ⚠️ 'batched inference' via vLLM | Throughput without a queueing penalty — static batching would blow the budget |
| KV / prefix cache | ⚠️ One clause | 🔴 The mechanism the budget depends on — 1,300 ms → 10 ms |
| Session affinity | 🔴 Absent — and contradicted | Required for prefix reuse to work at all |
| Quantization (INT8) | ✅ For memory and cost | Frees memory for batching and warm prefixes |
| Response cache | ✅ Emphasized | ⚠️ Low hit rate on inline completion (Lesson 3) |
| Degrade to silence | ✅ And correct | An error interrupts; silence does not |
| Speculative decoding | 🔴 Absent | Code is its best case — highly predictable tokens |
| Request cancellation | 🔴 Absent | Frees GPU capacity when the context goes stale |
| Model tiering | 🔴 Absent | The most frequent requests are the cheapest to serve |
Key takeaway
Naming vLLM is naming continuous batching, which is what lets a latency-critical service batch at all — static batching would consume the budget before inference began. The design's largest under-statement is KV cache, mentioned as a throughput feature when it is the mechanism that turns 1,300 ms of prefill into 10 ms — and it carries a consequence the chapter contradicts: prefix state lives on one GPU node, so routing must be session-affine, not round-robin across clusters. A stateless-looking inference tier that depends on KV reuse is not stateless. Retained prefixes and batch size compete for the same GPU memory, which is the stronger argument for quantization — it buys latency indirectly by freeing memory both optimizations consume. Degrading to silence rather than an error is right, and it makes retry and queue-for-later actively harmful under a budget set by changing input. Missing: speculative decoding, request cancellation, and model tiering for the most frequent class of request.
Next: what leaves the developer's machine.