Context Assembly

In one line: Lesson 3 showed that every token of context costs prefill time. This lesson is about spending that budget well — and about one technique the chapter omits that changes what "context" even means here.

The pipeline

The system uses a context extractor within the IDE to truncate and rank relevant snippets (for example, using Jaccard similarity or BM25) before sending them to the LLM.

To improve relevance, the system may enrich this prompt using a retrieval layer, which queries a vector database of repository embeddings. This is especially useful for large codebases, where important context may not be present in the currently open files.

Ranking on the client

Jaccard and BM25 are chosen because they run locally, and that constraint is the whole point

Neither is a sophisticated relevance model. Both are chosen for a property that matters more here than accuracy: they run in the IDE, in microseconds, with no network call and no model.

Jaccard similarity:  set overlap between token sets
                     |A ∩ B| / |A ∪ B|
BM25:                keyword relevance, weighted by term rarity

Compare with the alternative:

Embedding-based ranking:  better relevance, needs a model call
                          -> a round trip inside a 300 ms budget -> impossible
Jaccard / BM25:           "good enough", microseconds, on-device

Under a tight budget, a cheap approximation that runs where the data already is beats an accurate one that requires a round trip. That is the same conclusion the typeahead chapter reached about client-side debouncing and local caching, and that building block reached about doing personalization where the personal data lives.

There is a second reason that the chapter does not state and that matters for Lesson 7: ranking on the client means the code never leaves the machine to be ranked. Only the selected snippets are sent. A server-side ranker would require uploading everything you might want to rank — which for a code assistant is the entire repository.

Client-side ranking is a privacy mechanism as well as a latency one.

What the ranker should prefer, and why proximity is the wrong default

The design says "rank relevant snippets" without saying what relevance means. In a code file, the naive answer — the lines nearest the cursor — is frequently wrong.

The signals that actually matter, roughly in order:

1. Type and function DEFINITIONS for symbols visible at the cursor
2. IMPORTS — they say what vocabulary is in scope
3. The enclosing function or class
4. Recently EDITED regions (the developer's attention is there)
5. Sibling implementations — the file's existing patterns and conventions
6. Lines immediately above the cursor

The first is the highest-value and the hardest: knowing the shape of User matters more than the fifty lines above the cursor, and it usually lives in another file.

Note that the IDE already knows this. The language server has the symbol table, so "which definitions are in scope here?" is a question the editor can answer exactly, for free, without similarity heuristics.

Jaccard/BM25:      "which snippets look textually similar?"  -> approximate
Language server:   "which symbols are actually in scope?"    -> EXACT

The strongest context signal in an IDE is the one the IDE already computes, and the design never mentions using it. That is the cheapest available quality improvement in the chapter: combine exact structural signals from the language server with lexical similarity for everything else.

The same applies on the way out, per Lesson 1: a suggestion referencing a symbol that does not exist can be filtered before display, using the same symbol table.

Retrieval for large repositories

The retrieval layer earns its place, and it has a maintenance cost the design skips

The justification is correct: "important context may not be present in the currently open files." In a large codebase the function you need was written by someone else two years ago and you have never opened it.

Two things make code retrieval different from the document retrieval in that building block.

Chunking has natural boundaries. Prose has to be split on approximate semantic boundaries; code has functions, classes, and modules — chunk on those and every chunk is a self-contained, meaningful unit. This is the one place code is easier than prose.

Structural relationships beat similarity. Prose retrieval has only semantic proximity. Code has a call graph, an import graph, and a type hierarchy — and "the definition of the type I am using" is a far stronger signal than "a chunk that embeds nearby." Production systems combine both.

The cost the design does not mention: the index must track a repository that changes constantly.

Every commit, every branch switch, every local edit
  -> embeddings for changed files go stale
  -> and a stale code index is worse than a stale document index,
     because it suggests APIs that no longer exist

That implies incremental re-embedding on file save or commit, and it implies per-repository, per-branch index scoping. It also runs straight into Lesson 7: an embedded index of a private repository is a copy of that repository, on your infrastructure, subject to the same retention concerns as the code itself.

Code retrieval is easier to chunk and harder to keep fresh than document retrieval.

Fill-in-the-middle

Code completion has context after the cursor, and nothing in the design uses it

This is the modelling detail that most distinguishes code completion from text generation, and it appears nowhere in the chapter despite cursor_position being an API parameter.

A language model generates left to right: given a prefix, predict what comes next. But a developer's cursor is almost never at the end of the file.

python

def process_orders(orders):
    total = 0
    for order in orders:
        # <- CURSOR IS HERE
    return total          # <- the model needs to know this exists

The suffix is enormously informative. return total tells the model that the loop must accumulate into total. Without it, the model is guessing at a function whose ending it cannot see.

Fill-in-the-middle (FIM) is the technique that solves this. The model is trained with examples restructured so that it learns to condition on both sides:

STANDARD:  [prefix] -> predict continuation
FIM:       [prefix] [SUFFIX-MARKER] [suffix] [MIDDLE-MARKER] -> predict the middle

At inference the prompt is assembled with special tokens delimiting prefix and suffix, and the model generates the span between them.

This is not exotic — it is how production code models are trained, and it is the reason cursor_position is a parameter rather than an afterthought. The chapter passes cursor_position and describes its use only as an input to truncation.

Three consequences for the design:

Context assembly must preserve both sides of the cursor, not just the preceding lines — so the truncation strategy is "keep N tokens before and M tokens after," not "keep the last N tokens."

The prompt format is model-specific. FIM markers differ between model families, which couples the context service to the model version — a real deployment constraint the model registry should record.

It changes what "prefix" means for KV caching. With FIM, editing in the middle of a file changes the suffix portion of the prompt, which sits after the cached prefix — so prefix reuse still works, and this is a happy interaction rather than a conflict.

A code completion model must see the code after the cursor, and fill-in-the-middle is the mechanism — naming it is the clearest signal that you understand this problem as code completion rather than as generic text generation.

Spending the budget

Context source	Value	Prefill cost	Verdict
Lines around the cursor (both sides)	Essential — and the suffix needs FIM	Low	✅ Always
Imports	High — establishes the vocabulary in scope	Very low	✅ Always
Type/function definitions in scope	Highest — and the language server knows them exactly	Medium	✅ The underused signal
Open tabs, ranked	Medium — proxy for developer attention	Medium–high	⚠️ Rank hard, include few
Retrieved repository snippets	High for large codebases	High	⚠️ Few, and only when the local signal is weak
Whole files	Marginal beyond the ranked parts	Prohibitive	🔴 Never

The context budget should be adaptive, not fixed

The design treats truncation as a fixed operation — fit the prompt to the window. Lesson 3 showed the real constraint is prefill latency, and that suggests something better.

Vary the context budget by what the prefix cache can absorb. If the developer is editing in the same file and the prefix is cached, the marginal cost of context already sent is roughly zero — so you can afford to have sent a lot. If they just switched files, everything is a cold prefill and the budget is tight.

WARM (same file, prefix cached):  spend generously — the cost is already paid
COLD (file switch, first request): spend minimally — every token is 0.33 ms

Similarly, vary by request type. Inline completion is under 300 ms and must be lean. explainCode is user-initiated, the developer expects a pause, and it can afford far more context.

getCompletion:  tight budget, lean context, TTFT-critical
explainCode:    generous budget, rich context, seconds are acceptable

One context strategy for two latency contracts is a compromise that serves neither — and the design applies the same assembly path to both APIs.

Key takeaway

Ranking with Jaccard or BM25 on the client is right because they run in microseconds with no round trip — and it is a privacy mechanism as well as a latency one, since only the selected snippets ever leave the machine. The design's largest missed signal is that the IDE already knows exactly which symbols are in scope: combining the language server's structural facts with lexical similarity is the cheapest quality win available, and the same symbol table can filter hallucinated identifiers before display. Code retrieval is easier to chunk than prose — functions and classes are natural boundaries — and harder to keep fresh, since a stale index suggests APIs that no longer exist. The chapter's clearest omission is fill-in-the-middle: a cursor has code after it, the suffix is enormously informative, and FIM is how production code models use it — which is the real reason cursor_position is an API parameter. Finally, the context budget should adapt to whether the prefix is cached and to which API is being served.

Next: the APIs and the storage schema.