Caching: Three Tiers
In one line: three different caches sit at three different points, they save different things, and only one of them can return an answer to a question nobody asked.
The three, and what each saves
| Exact cache | Prefix cache | Semantic cache | |
|---|---|---|---|
| Keys on | The full request, byte-identical | A shared token prefix | Embedding similarity to a past request |
| Saves | The entire generation | Prefill only, not decode | The entire generation |
| Lives | In front of the model | Inside the engine, in GPU memory | In front of the model |
| Hit rate | Low — exact repeats are rare | Very high — system prompts repeat always | Moderate to high |
| Risk | Staleness only | None — it is a pure optimisation | Returning the wrong answer confidently |
The columns are ordered by how much you should trust them, and that ordering is the lesson.
Prefix caching is the free one
Every request carries a system prompt, and in a retrieval system it carries retrieved context too. Those tokens are processed on every request even though the model has seen them a thousand times.
Prefix caching keeps the processed KV state for a shared prefix and reuses it, so only the new suffix is prefilled.
Two design consequences follow, and both are cheap to state and easy to miss.
**Put the stable part of the prompt first. A cache keyed on a prefix only matches from the beginning, so a request that starts with a per-user timestamp shares nothing with the next request. Ordering the prompt stable-to-variable is a one-line change with a large effect.
It only helps prefill. Decode still runs in full. So prefix caching improves time to first token and throughput, and does nothing for a request whose cost is dominated by a long generation.
This is also the mechanism that makes prefix-aware routing worth doing — the cache exists per worker, so sending a request to the wrong worker discards a hit that was available.
Exact caching is safe and rarely hits
Key on a hash of the complete request; return the stored response on a match. Trivially correct, and in a conversational product almost nothing repeats byte-for-byte.
It earns its place in narrower cases: a fixed set of suggested prompts, an autocomplete over a bounded vocabulary, an internal pipeline that reprocesses the same documents, or an evaluation harness that runs the same suite repeatedly. That last one is worth remembering — evaluation runs are real inference spend, and they are highly cacheable.
Semantic caching is powerful and dangerous
Embed the incoming request, search for a past request within a similarity threshold, and return its stored answer.
The hit rate is far higher than exact caching, because paraphrases are common. And the failure mode is the sharpest in this chapter: two requests can be extremely close in embedding space and have different correct answers.**
This is the first lesson of the retrieval chapter arriving with consequences: embeddings encode topic strongly and specifics weakly, so the distinguishing detail — a date, a number, an identifier, a negation — is exactly what the similarity score is worst at seeing.
Making it safe
The mitigations are all about narrowing where it applies rather than tuning the threshold alone.
**Scope the key. Include user, tenant, locale and anything else that changes the correct answer, so a hit can only come from a genuinely comparable context.
Do not cache anything personalised or time-sensitive. If the answer depends on who is asking or when, semantic caching is unsafe by construction.
Set the threshold empirically, and high. Take labelled pairs, measure how often a hit at a given similarity would have returned a wrong answer, and choose accordingly. A conservative threshold with a lower hit rate is the right default.
Measure the harm, not the hit rate.** The tempting metric is cache hit rate, and it rewards exactly the wrong behaviour — loosening the threshold always improves it. Sample hits and check whether the served answer was actually correct for the new question.
Invalidation
The usual hard problem, with an extra wrinkle: a cached answer can go stale because the world changed, not because anything in your system did.
Three mechanisms, and most systems need more than one:
**Time to live. Blunt and effective. Short for anything current, long for stable reference material.
Event-driven invalidation. When a source document changes, drop the cached answers derived from it. This requires tracking which retrieved chunks contributed to which cached response — worth doing in a retrieval system, and it is provenance you probably want anyway.
Version keying.** Include the model version, prompt version and index version in the cache key. Then a deploy invalidates cleanly and automatically rather than serving answers from the previous configuration.
That third one is the one teams forget, and its symptom is confusing: a prompt improvement ships and quality does not move, because most traffic is being served from a cache populated by the old prompt.
Key takeaway
Three caches at three points: prefix caching is a pure optimisation inside the engine that saves prefill only, so put the stable part of the prompt first; exact caching is safe and rarely hits outside bounded or repeated workloads like evaluation runs; and semantic caching has the highest return and can serve a confidently wrong answer, because embeddings see topic clearly and the distinguishing detail poorly. Scope its key, exclude anything personalised or time-sensitive, and measure served-answer correctness rather than hit rate — which is a metric that rewards loosening. Version the cache key on model, prompt and index, or a deploy will keep serving the old configuration.
Next: choosing which model handles the request at all.