Semantic Caching

In one line: this is the first component in the course whose cache can return something different from what a miss would have produced. Everything interesting about it follows from that.

Is a typical cache (like LRU or TTL-based) sufficient for storing AI-generated responses?

Why the classic answer fails

While traditional caching strategies such as LRU or TTL are effective for static or frequently accessed web content, they are often inadequate for AI-generated responses, which are highly contextual and dependent on nuanced input factors such as prompt wording, user history, and personalization.

The failure is not the eviction policy — it is the key

Read carefully: LRU and TTL are eviction policies. They decide what to throw away. They are perfectly fine at that job here.

What fails is the assumption underneath every conventional cache: that there is a key, and that identical keys mean identical answers.

Consider three prompts:

"How do I reverse a linked list in Python?"
"python code to reverse a linked list"
"Show me how to reverse a linked list using Python"

An exact-match cache treats these as three unrelated keys and generates three separate responses — three times the GPU cost from Lesson 4. Yet they want the same answer, and a human would say so instantly.

The reverse failure is worse. Consider:

"Summarize the document"

Two users send byte-identical prompts. Under an exact-match cache they collide — and they must not, because "the document" means different things to them. The prompt text is not the whole input; conversation history and user profile are also inputs, and they are invisible to the key.

So exact-match caching fails in both directions at once: it misses on prompts that mean the same thing and hits on prompts that do not. That is not a tuning problem. It is a signal that the key needs to be meaning, not bytes.

The semantic cache

To address this, ChatGPT-like systems benefit from a semantic cache that stores not only the response but also a vector embedding of the input prompt that captures its meaning. When a new prompt is received, the system can retrieve previous responses whose embeddings are semantically similar using a vector-based cache powered by approximate nearest neighbor (ANN) search. This enables the reuse of responses based on meaning rather than exact text matches.

What an embedding is, in one paragraph

An embedding maps text to a fixed-length list of numbers — typically several hundred to a couple of thousand — positioned so that texts with similar meaning land near each other. "Reverse a linked list in Python" and "python code to reverse a linked list" produce vectors that are close; "what's the weather in Paris" produces one far away.

That gives a numerical measure of semantic similarity, usually cosine similarity, where 1.0 is identical in direction and 0 is unrelated.

Note that the embedding step is not extra machinery invented for the cache. Lesson 2 argued that generating embeddings is the honest modern justification for the pre-processing NLU service, and the vector database in the detailed design needs them too. One embedding computation serves the cache lookup, the retrieval step, and any routing decision — which matters, because embedding is itself a model call, just a very small and cheap one compared to generation.

Approximate nearest neighbour — and why approximate is the point

Finding the closest vector exactly means comparing against every stored vector. With millions of cached entries on the critical path of every request, that is hopeless.

ANN trades exactness for speed. Structures like HNSW graphs or IVF partitions find a very close neighbour in logarithmic rather than linear time, occasionally missing the true nearest.

The trade is comfortable here for a specific reason worth stating: a cache miss is already a correct outcome. If ANN fails to find a cached entry that existed, we generate the response — which is exactly what we would have done anyway. The cost is a wasted GPU-second, not a wrong answer.

That asymmetry is why approximation is acceptable in a cache and would not be in, say, a payments ledger. When one error direction is free and the other is expensive, design the approximation to fail in the free direction.

The Distributed Search chapter made the same argument about index recall, and that building block made it about pruning segments with aerial distance. Same shape: an admissible approximation that can only cost you work, never correctness.

The threshold is the whole design

Everything about a semantic cache reduces to one number

A hit requires similarity above some threshold. That number determines the system's behaviour entirely, and both directions are dangerous.

Set it too low and you get false hits — semantically near, materially different prompts colliding:

"What is the capital of Australia?"     -> Canberra
"What is the capital of Austria?"       -> similarity is very high

Two country names, one letter apart, embeddings close together — and returning "Canberra" for Austria is a confidently wrong answer the user has no way to detect. A false hit in a semantic cache is not a stale value; it is a fabricated one.

Set it too high and you approach exact matching and lose the savings that motivated the cache.

Three things make this manageable, and naming them is what separates a real answer from a repeated definition:

Scope the cache narrowly. Cache only prompts with no conversation history and no personalization — first-turn, generic questions. Those are the ones where a shared answer is legitimately correct, and there are a lot of them at 150 million daily users.

Include the context in the key. If history matters, embed the assembled context rather than the bare prompt. Two users asking "summarize the document" then produce different vectors, because their documents differ.

Make the threshold a per-domain decision. Factual lookups, where near-miss confusions like Australia and Austria live, need a high bar. Open-ended requests like "write a poem about autumn" tolerate a much lower one, because approximate reuse is genuinely fine.

What a modest hit rate is worth here

Conventional cache discussions chase 90% hit rates, because a miss costs a database round trip and the ratio of hit cost to miss cost is maybe a hundred to one.

Here the ratio is different by orders of magnitude:

So even a 5% hit rate removes 5% of a ten-thousand-GPU fleet — hundreds of GPUs, which is real money. A hit rate that would be embarrassing for a web cache is valuable here.

This is the concrete form of the point from Lesson 5: when the cached item is expensive to produce rather than slow to fetch, the economics of caching change and low hit rates start paying for themselves.

Prefix caching — the other cache, and often the bigger win

Semantic caching reuses whole responses. There is a second, less obvious reuse available, and in production it frequently saves more.

Recall from Lesson 6 that every prompt is assembled with a shared prefix — system instructions, policy, persona — identical across millions of requests. And within a conversation, turn five contains all of turns one through four verbatim.

Prefill has to process all of that. But the KV cache entries for a shared prefix are identical for every request that shares it, so they can be computed once and reused.

Prefix caching does exactly that. It cuts prefill work — and therefore time to first token, the metric users feel most — for two very common cases:

• The system prompt, shared by everyone.
• Conversation history, where turn N reuses everything computed for turn N-1.

The second is why multi-turn conversations do not get linearly slower as they grow, despite the context growing every turn.

Note the difference in what the two caches store. The semantic cache holds outputs and risks returning a wrong answer. Prefix caching holds intermediate state for a prefix that is byte-identical, so it is exactly correct — a pure saving with no accuracy risk. If you can only mention one in an interview, mention this one.

Interview signal by level

Next: the full component inventory.