Query Transformation

In one line: you are embedding a short question and matching it against long declarative passages, and closing that shape mismatch is often worth more than any index tuning.

The asymmetry

A user asks "why is my build slow?". The passage that answers it says "Build times increase when the dependency cache is cold, because each module is recompiled from source."

They share almost no vocabulary. One is short, interrogative and vague; the other is long, declarative and specific. Embedding models are trained to handle some of this, and the gap remains real.

The conversational query is worse

In a multi-turn product, most queries are not self-contained at all.

User: How do I rotate an API key?
Bot:  [explains rotation]
User: And for service accounts?

Embedding "And for service accounts?" retrieves documents about service accounts generally — not about rotating their keys. The subject lives in the previous turn, and retrieval only sees this one.

This is the single most common retrieval failure in conversational products, and the fix is mandatory rather than optional: rewrite the query into a standalone form before retrieving.

"And for service accounts?"
  + conversation history
  -> "How do I rotate an API key for a service account?"

A small model does this well and cheaply. The cost is one extra call in the latency path, which is why it belongs in the time-to-first-token budget from the serving chapter.

Four transformations, and what each is for

Decomposition

"How does our pricing compare to the enterprise tier?" needs two different passages. A single embedding lands somewhere between them and may retrieve neither well.

Split it, retrieve for each part, and merge:

The cost is more retrieval calls and a larger context. The benefit is that compound questions stop returning half an answer — a failure mode that is otherwise very hard to diagnose, because the answer looks complete.

Hypothetical answers

Rather than embedding the question, have a model write a plausible answer and embed that. The fake answer is declarative and long, so it looks like the documents — matching passage to passage rather than question to passage.

It can be genuinely effective, and it has two honest costs. It adds a generation call before retrieval, which is significant in the latency budget. And the hypothetical answer can be confidently wrong in a way that steers retrieval away from the truth — a fabricated answer about a feature that does not exist retrieves documents about things that do not matter.

Treat it as a technique to reach for when measurement shows the shape mismatch is your bottleneck, not as a default.

Query routing is part of this

Transformation assumes retrieval is the right move. Often the first decision is whether to retrieve at all.

The aggregation branch is the one from the first lesson, and routing is where it gets handled. Retrieving five documents for "how many customers mentioned pricing" produces a precise-sounding number from a sample the user never knew about.

Skipping retrieval for conversational turns is also worth a line: it saves latency and cost, and it avoids the model being handed irrelevant context it may feel obliged to use.

Next: retrieval and reranking, and choosing how much to pass on.