Context and Retrieval

In one line: the design's conclusion names this as the biggest gap in its own design — "we discussed the context-awareness module, but how is that actually created and integrated?" This lesson answers that question, because everything the product feels like depends on it.

The context window is a budget

Every model has a maximum number of tokens it can process in one call — the context window. Everything the model knows about the current request must fit inside it.

[system instructions]          ~500 tokens    fixed
[user profile and preferences] ~200 tokens    small
[conversation history]         ~??? tokens    GROWS EVERY TURN
[retrieved documents]          ~??? tokens    you choose
[the current prompt]           ~500 tokens    given
--------------------------------------------
                        must fit in the window

Two things make this a budget rather than a limit

The first is obvious: history grows every turn while the window does not. A long conversation eventually exceeds it, and something must be dropped.

The second is the one people miss, and it is the reason "just use a bigger window" is not an answer.

Every token in the window costs money and latency. From Lesson 4:

• Prefill processes every input token, so context length directly sets time to first token.
• KV cache scales linearly with context length, and KV cache is what limits how many users share a GPU.

Fill a 128k-token window and you have not merely used a feature — you have made that request's prefill enormous and consumed KV space that could have held dozens of other users' sequences.

So context is a resource with a real per-token price, paid on every single request. That is why retrieval exists at all. If context were free, you would include the entire conversation and the entire knowledge base and never think about it again. The engineering exists precisely because it is not.

The design goal is not maximum context. It is the smallest context that produces a good answer.

How the system fills the budget

Four sources, each with a different strategy:

Recent turns verbatim, older turns by retrieval — and why that split is right

The two mechanisms in this design are not competing. They handle different regimes.

Recent turns are included whole, because in dialogue the immediately preceding exchange is almost always relevant. The pronoun problem from Lesson 1 — "what else has she written?" — is resolved by the turn right before it, not by a similarity search. And prefix caching means the earlier turns of the conversation are already computed, so including them is cheap.

Older content is retrieved, because it is mostly irrelevant. Something said forty turns ago is usually noise, but occasionally decisive. Including it all is unaffordable; including none of it makes the system forgetful. Retrieval picks the few pieces that matter.

The boundary between the two is a tuning decision, and it is exactly the kind of thing worth naming: recency is a cheap approximation of relevance that works well nearby and badly far away, so use it where it works and pay for similarity search where it does not.

Retrieval-augmented generation

The same machinery that recalls old conversation turns also grounds answers in documents the model was never trained on.

Retrieval solves three problems a bigger model cannot

Worth being precise about what retrieval is for, because "it gives the model more information" undersells it.

Knowledge the model does not have. Training data has a cutoff, and it never contained your company's internal documents. No model size fixes this — the information was simply not there.

Attribution. A retrieved chunk has a source, so the response can cite it and a user can check. A claim generated from weights has no provenance whatsoever. This is the closest thing the design has to an answer for the accuracy gap from Lesson 2 — not a solution, but the difference between a checkable claim and an unfalsifiable one.

Update without retraining. Correcting a fact means updating a document and re-embedding one chunk — minutes. Correcting knowledge baked into weights means retraining — weeks and enormous cost.

That third point is the strongest architectural argument. It cleanly separates capability, which lives in the weights and changes slowly, from knowledge, which lives in the index and changes continuously.

It is the same instinct as that building block's dual-weight scheme: precompute the stable part, and keep the volatile part in a layer you can change cheaply. There, base weights versus transitory traffic. Here, model weights versus retrieved documents. Different domain, same structural move.

Chunking is where retrieval quality is won or lost

Documents must be split before embedding, because an embedding of an entire 50-page manual is too diffuse to match anything usefully. But the split is not innocuous.

Chunks too small — a sentence — and each is embedded without the context that gives it meaning. The chunk "this is not recommended for production use" is useless when you cannot tell what "this" refers to.

Chunks too large — a chapter — and the embedding averages several topics into a vector that matches nothing strongly, and you spend context-window budget on mostly irrelevant text.

The pragmatic answers all involve preserving structure: split on semantic boundaries like sections and paragraphs rather than fixed character counts, overlap consecutive chunks so a boundary does not sever a thought, and prepend headings to each chunk so it carries its own context.

The general point is worth extracting because it recurs: the retrieval unit and the reasoning unit are different sizes, and the mismatch is the whole difficulty. You retrieve fragments and reason over documents.

More retrieved context is not better — three ways it degrades

The tempting reflex is to retrieve twenty chunks instead of five, on the theory that more information cannot hurt. It can, in three distinct ways.

Cost and latency. Twenty chunks is thousands more prefill tokens on every request, more KV cache, fewer concurrent users. Straight from Lesson 4.

Attention dilution. Models attend less reliably to material in the middle of a long context than at its beginning or end. Burying the one relevant chunk among nineteen irrelevant ones can make it less likely to be used.

Contradiction. Retrieve enough documents and some will disagree — an old policy and its replacement, two versions of a specification. The model has no reliable way to adjudicate and may confidently follow the wrong one. Retrieving more raises the chance of retrieving something wrong.

The mitigations are ordering rather than volume: rerank the retrieved set with a more expensive model that scores relevance properly, keep only the top few, place the most relevant material at the edges of the context rather than the middle, and prefer recency when sources conflict.

The general principle: precision matters more than recall in retrieval for generation. That is the opposite of a search engine, where showing the user ten results and letting them choose is fine. Here the model cannot ignore a bad result — it reads everything you give it.

Per-user isolation is the failure that ends careers

Lesson 8 flagged that the vector database holds both shared knowledge and per-user conversation history. This is where that matters.

If retrieval runs against a single index without a hard user filter, one person's query can surface another person's conversation — and the model will helpfully weave it into a fluent answer.

The safeguards are unglamorous and non-negotiable: separate indexes per tenant where possible, mandatory user or tenant filters applied at the query layer rather than left to callers, and tests that specifically attempt cross-user retrieval.

It is worth raising unprompted in an interview. It is the most likely serious bug in this architecture, it is invisible in the diagram, and it is exactly the kind of thing a system that "works" in testing does wrong in production.

Interview signal by level

Next: the feedback loop and how usage becomes a bill.