Context Assembly

In one line: the same chunks arranged differently produce different answers, because a model does not attend to a long context evenly.

Lost in the middle

Language models show a U-shaped performance curve over position: they use information best at the beginning and the end of the context, and worst in the middle.

The finding that makes it concrete: when the relevant document was placed in the middle of a long context, a model's accuracy on a multi-document question dropped below its accuracy with no documents at all. Retrieval did not merely fail to help — it actively hurt.

Two design consequences follow directly.

Order by relevance, outward from the edges. The reranker gave you a ranking — use it. Put the top passage first, then the next-best last, and let the weakest sit in the middle where they matter least. This costs nothing and is a genuinely free improvement.

Longer context is not automatically better. Performance degrades as context grows even in models built for long inputs, which reinforces the previous lesson: keep the final k tight. A model with a 200K window is not a reason to send 200K tokens.

Deduplicate first

Overlapping chunks and near-duplicate documents mean the same content often arrives several times — versioned copies of a page, a passage repeated across a FAQ and a guide, or simply the overlap you deliberately built into chunking.

Duplicates cost twice: tokens you pay for, and repetition that can make the model treat a claim as better-supported than it is.

Deduplicate on content similarity rather than exact match, since near-duplicates are the common case, and keep the highest-ranked instance.

The token budget

Context is a fixed budget with competing claimants, and writing it down forces the trade-offs into the open.

System prompt and instructions        400
Conversation history                  800   <- grows every turn
Retrieved chunks                    2,500   <- the variable you control
Room for the answer                 1,000
                                   ------
Total                               4,700

Two allocation rules worth stating:

History and retrieval compete. As a conversation lengthens, history consumes budget that retrieval needs. Something must give, and deciding what in advance is better than truncating whatever happens to be last. Usually the answer is to summarise history past some depth rather than trim retrieval.

Reserve room for the output. A context filled to the limit leaves no space to answer, and the failure is a truncated response rather than an error.

Truncate deliberately

When it does not fit, something is dropped. Doing it in the middle of a chunk is the worst option — the model receives a passage that stops mid-sentence and may still cite it.

Drop whole chunks from the bottom of the ranking, and record that it happened. A truncation flag in the logs is what later explains a bad answer that looks inexplicable.

Make citation possible

Attribution was one of the three reasons to choose retrieval at all, and it has to be built into the prompt rather than hoped for.

Label chunks with short stable identifiers, include the document title and section so the model can refer to them naturally, and require a citation per claim.

The verification step is the one that matters. A model can cite an identifier that exists and does not support the claim — a plausible reference to real source material that says something else. Checking that the cited chunk was actually retrieved is free; checking that it supports the claim is the faithfulness measurement from the evaluation chapter.

Structure the prompt for caching

One serving consequence worth carrying over. Prefix caching only matches from the beginning of the prompt, so the layout should run stable to variable:

1. System prompt and instructions   <- identical every request, cacheable
2. Retrieved chunks                 <- varies by query
3. Conversation history             <- varies
4. The user's question              <- always new

Putting a timestamp or a user id at the very top destroys prefix cache reuse for every request in the system. It is a one-line ordering decision with a measurable effect on time to first token and cost.

Next: getting the model to actually use what it was given.