Where the 300 Milliseconds Goes

The property that makes this unlike a chat assistant: a late suggestion is not a slow suggestion, it is a discarded one. There is no partial credit.

The budget

Cross-region network alone can spend the entire budget

Start with the stage nobody controls.

Same region, warm connection      RTT   30–50 ms
Same continent, warm              RTT   50–100 ms
Cross-region (e.g. EU to US-east) RTT  100–250 ms   <- the budget, gone
Cold TLS handshake                2–3 RTTs, so 100–150 ms on top

A single cross-region round trip can consume 100% of the 300 ms, leaving nothing for the inference the request was made for.

That is why the chapter's line — "regional deployment is essential... keeping requests close to developers" — is not an availability nicety. It is the only way the budget closes. GPU capacity in five regions is expensive and it is not optional.

Two consequences the design does not state:

Connections must be kept warm. A cold TLS handshake costs two to three round trips before the request body moves. At 300 ms total, that is fatal. The IDE plugin must hold a persistent connection — which is a second argument for SSE or WebSockets beyond streaming, and it is the same conclusion the typeahead chapter reached about warming connections.

The debounce is outside the budget and is the largest single delay. The design's practical tip suggests 150–200 ms of client-side debounce before firing a request. From the developer's perspective that is part of the wait — so the felt latency is debounce plus everything else, and the real end-to-end figure is closer to 450–500 ms than 300.

A latency budget that excludes the client-side delay measures the server, not the experience.

Prefill is the dominant server-side cost

Time-to-first-token is prefill time, and this system's prompts are large by design

Lesson 2 established the ratio: 3 KB in, 1 KB out. This is an input-heavy workload, which is the inverse of the support bot — and it determines which half of inference matters.

PREFILL: process the whole prompt, produce the first token
         -> parallel over the prompt, but scales with PROMPT length
         -> THIS IS TTFT

DECODE:  generate each subsequent token
         -> sequential, scales with OUTPUT length
         -> hidden by streaming

Since the output is a few lines of code, decode is short and streaming hides it. The entire 300 ms problem is prefill.

And prefill scales with the thing this system is built to send:

Prompt at ~3,000 tokens/s prefill throughput:

  1,000 tokens (a small window)      ->   333 ms   <- ALREADY over budget
  2,000 tokens                        ->   667 ms
  4,000 tokens (file + open tabs)     -> 1,333 ms
  8,000 tokens (+ retrieved snippets) -> 2,667 ms

A 4,000-token context is over a second of prefill. The 300 ms budget implies a prompt on the order of a few hundred tokens on a cold path — which is far less context than "current file, open tabs, imports, project structure" suggests.

This is the tension the chapter never surfaces, and it is the same shape as the support bot's: the features that make suggestions good are the features that make them slow. More context means better completions and longer prefill.

The chapter says context is "truncated to fit within the model's context window." The window is not the binding constraint — prefill latency is, and it binds an order of magnitude earlier.

In an input-heavy LLM workload, the context budget is set by prefill latency, not by the context window. That reframing is what makes the next lesson's ranking work matter, and it is why the following callout's optimization is the most important one in the chapter.

Prefix caching is what actually closes the budget

Consecutive keystrokes share almost their entire prompt — and the design mentions this in passing

Here is the property that rescues the arithmetic above, and the chapter names it once, as a clause under vLLM: "leveraging KV cache optimization."

Consider what changes between two consecutive completion requests from the same developer:

Request at t:      [4,000 tokens of context] + "def process_"
Request at t+1s:   [4,000 tokens of context] + "def process_data"
                    ^^^^^^^^^^^^^^^^^^^^^^^
                    IDENTICAL — the file barely changed

The prompts share a prefix of thousands of tokens. If the server retains the computed attention state (the KV cache) for that prefix, prefill for the next request only processes the delta:

WITHOUT prefix reuse:  prefill 4,000 tokens  -> ~1,300 ms   -> budget blown
WITH prefix reuse:     prefill    ~20 tokens ->    ~10 ms   -> budget closes

Two orders of magnitude, and it converts an impossible budget into a comfortable one.

That makes prefix caching not an optimization but the mechanism the whole design depends on, and it has architectural consequences the chapter does not draw:

Requests from one session must route to the same GPU node, or the cache is on the wrong machine. That means session affinity in the load balancer, which is in tension with the design's stated "distribute traffic across GPU clusters."

KV cache is GPU memory, and it competes with batch size. Retaining prefixes for many sessions consumes exactly the memory that batching needs — a real capacity trade the design never mentions.

Cache eviction is now latency-visible. A developer whose prefix was evicted pays the full 1,300 ms prefill, so tail latency is governed by eviction policy.

When consecutive requests share most of their prompt, prefix KV reuse is the difference between a viable and an impossible latency budget — and it turns a stateless inference tier into a session-affine one.

Response caching on a prompt hash is nearly useless here, for a reason the design does not notice

The chapter specifies a completion cache keyed on "a hashed representation of the assembled prompt." It then quotes plausible hit rates:

Even modest cache hit rates (for example, 10–20%) can significantly reduce GPU inference load at scale.

For inline completion, the achievable rate is far lower, and the reason is structural: the prompt contains the file content and the cursor position, both of which change on every keystroke.

Keystroke N:   hash(context + "def process_")     -> key A
Keystroke N+1: hash(context + "def process_d")    -> key B, MISS
Keystroke N+2: hash(context + "def process_da")   -> key C, MISS

Every request is a new key. Exact-match hits require a different developer, in a different file, to arrive at a byte-identical context — which happens for boilerplate (public static void main, standard imports, common test scaffolding) and essentially nowhere else.

So the 10–20% figure is plausible in aggregate across all traffic, driven by the boilerplate tail, and close to zero for the interactive completions that define the product.

The distinction matters because the two caches do different jobs:

RESPONSE cache (prompt hash):  helps the BOILERPLATE tail       -> low hit rate
PREFIX KV cache:               helps EVERY interactive request  -> the real win

Cache the computation, not the answer, when the input changes on every request. The design has both mechanisms and emphasizes the weaker one.

The TTL warning the chapter gives is right, incidentally — "too low reduces the hit rate, too high serves outdated suggestions" — and it applies to the response cache, where a completion cached against a since-changed file is actively wrong.

Speculative decoding

The technique that reduces generation latency, absent from the design, and unusually well suited to code

Not mentioned anywhere, and it belongs here because code is the ideal case for it.

A small draft model proposes several tokens ahead; the large model verifies them all in one forward pass. Accepted drafts are kept, and the first rejected one restarts the process.

NORMAL DECODE:  1 forward pass per token
SPECULATIVE:    draft 4 tokens cheaply -> verify all 4 in ONE pass
                -> if 3 accepted, you produced 3 tokens for ~1 pass

The speedup depends on how predictable the text is — and code is far more predictable than prose. Closing brackets, standard boilerplate, repeated identifiers, and conventional formatting are exactly the tokens a small model gets right.

Two variants are relevant here:

A small draft model trained on the same code corpus.

Prompt lookup decoding — even simpler, and strikingly apt: draft the next tokens by copying from the prompt itself. Code completions frequently repeat identifiers and structures already present in the file, so a plain string match often produces valid drafts with no model at all.

Speculative decoding reduces decode time, which streaming already hides — so its value here is smaller than prefix caching's. But it matters for explainCode, where the output is long, and it lowers cost per token across the board.

Speculative decoding pays off in proportion to how predictable the output is, which makes code its best case.

The budget, closed

The budget closes on the warm path only, which changes what you monitor

Read the two columns together. The design is not fast; it is fast when warm.

That has a direct consequence for how the system is measured, and it is the thing to say in an interview:

MEDIAN TTFT:  dominated by the warm path -> looks excellent
P99 TTFT:     dominated by COLD paths    -> 10-20x worse

Cold paths are not rare in practice. They occur on the first request of a session, after a cache eviction, when a developer switches files (the prefix changes wholesale), after a deploy flushes GPU caches, and on regional failover.

So the metrics that matter are not just latency percentiles but the things that cause the cold path:

Prefix cache hit rate         <- the single most important number
Cold-start TTFT specifically  <- measured separately from warm
Session affinity hit rate     <- did the request reach the right GPU?
Evictions per minute          <- rising evictions predict rising P99

When a system's performance depends on a cache, its latency SLO must be stated for the cold path, and the cache hit rate is a first-class SLI. Reporting only median TTFT hides the experience of every developer who just opened a file.

Next: assembling the context that prefill has to pay for.