Your RAG retrieves the wrong chunks? Fix the query first

Teams tuning a RAG pipeline usually spend their effort on three things: the chunking strategy, the embedding model, and a reranking pass. All three matter. But there is a step upstream of every one of them that most pipelines never touch: the query the user typed goes into the retriever exactly as written. That raw query is often the weakest link in the whole chain. It is too short, it is phrased as a question when your documents are written as answers, or it refers to something from three messages ago. Reshaping the query before it hits the retriever is one of the cheapest recall levers you have, because it changes nothing about your index.

• The user's question and the wording of the answer in your corpus rarely match. Embedding a question to find an answer leaves recall on the table.
• Query transformation (rewriting, expansion, multi-query, HyDE, decomposition) reshapes the query before retrieval, so you gain recall without re-chunking or re-embedding anything.
• Every technique costs an extra LLM call and some latency. Route by query type and gate the change behind a retrieval eval, or you will pay for reformulations that do not help.

Why the raw user query is a bad search query

A retriever compares the query against your chunks in the same vector space, so the query has to look like the thing you want to find. Real user queries rarely do. There are four recurring problems. First, vocabulary mismatch: the user asks "why did my payment bounce?" and the answer in your docs is titled "declined transaction error codes." Same meaning, different words, weak similarity. Second, length: a three-word query carries almost no signal for an embedding model, so the top results are close to random. Third, conversational references: "what about the second one?" is meaningless without the previous turn. Fourth, multiple intents packed into one sentence: "compare the Pro and Enterprise plans and tell me which supports SSO" is two retrievals wearing one query.

None of these are fixed by a better index. The chunk you need is sitting there; the query just does not point at it. That is what makes query transformation such a high-return change. You are not buying a bigger model or re-embedding a corpus, you are spending one cheap LLM call to turn a bad search query into a good one.

Rewrite and expand the query

The lightest technique is query rewriting: ask a small, fast model to rewrite the user's message into a clean, self-contained search query. That single step resolves pronouns, strips filler, and fixes obvious phrasing. It is the highest-value change per token because almost every downstream technique works better on a clean query than on a raw one.

Query expansion goes one step further by adding related terms. Instead of searching only "payment bounced," you also search "declined transaction," "failed charge," and "card error," then merge the results. Expansion helps most when your corpus uses precise domain vocabulary that users do not know. It costs one generation call and a slightly wider retrieval, and it is a good default for short or jargon-light queries.

Multi-query fan-out and fusion

Multi-query retrieval makes the "my first guess is incomplete" assumption explicit. You ask the model to produce several reformulations of the query, each capturing a different reading of the user's intent, run retrieval for all of them, and then fuse the result lists. The fusion step is where reciprocal rank fusion earns its place: a chunk that shows up near the top for several reformulations rises above one that ranked highly for a single phrasing but nothing else. The payoff is recall on genuinely ambiguous questions; the cost is N retrievals plus a dedup and fusion pass, so reserve it for queries where ambiguity actually hurts rather than running it on every request.

HyDE: search answers against answers

Hypothetical Document Embeddings (HyDE) attacks the vocabulary-mismatch problem head on. Instead of embedding the question, you ask the model to write a short hypothetical answer to the question, then embed that answer and use it as the retrieval query. It sounds strange, but it works because your corpus is written as answers, not questions, and an answer-shaped query lands closer to real answer chunks in the vector space. The generated answer can be factually wrong and HyDE still helps, because you are using its shape and vocabulary for matching, not its facts. HyDE shines on short or underspecified queries; the tradeoff is the extra generation latency on the critical path, so it is best applied conditionally rather than to every query.

Decompose multi-hop and conversational queries

Some queries need to be split before they are searched. A compound question like "which plans support SSO and what do they cost?" retrieves better as two sub-queries whose results are combined, because a single embedding of the whole sentence is a blurry average of two topics. Query decomposition asks the model to break the question into standalone sub-questions, retrieves each, and passes the union to the generator. It is the retrieval-side answer to the multi-hop problem.

Conversational contextualization solves the "second one" problem. Before retrieval, rewrite the latest user turn into a standalone query using the recent history, so "what about the second one?" becomes "what is the price of the Enterprise plan?" This is a small change that fixes a large and common class of silent retrieval failures in chat features, where the query sent to the retriever quietly loses the thread.

Choose per query, and measure the change

These techniques are not free and they are not additive by default. Each one adds an LLM call and latency, and stacking all of them on every request buys diminishing recall for real cost and slowness. Mature pipelines route by query shape: very short or vague queries get HyDE, ambiguous queries get multi-query fusion, compound queries get decomposition, and the ordinary majority get a cheap rewrite plus expansion. A tiny classifier or a heuristic on query length and structure is enough to pick.

Whatever you add, gate it behind a retrieval metric rather than vibes. The common failure is to ship a clever reformulation that looks smart in a demo and quietly lowers recall on the long tail. Track recall at k on a labeled query set before and after, the way the common RAG eval mistakes piece lays out, and keep only the transformations that move the number. Query transformation is a core part of a production RAG architecture, but like every other lever it earns its place only when the eval says it does.

Frequently asked questions

Is query rewriting better than reranking?

They fix different problems, so it is not either/or. Query transformation changes what you retrieve by improving the search query; reranking changes what you keep by reordering the candidates you already pulled. A strong pipeline does both: transform the query to widen and sharpen the candidate pool, then rerank to promote the best chunks in that pool. If retrieval misses the right chunk entirely, reranking cannot save you, which is why the query side comes first.

Does HyDE work if the model hallucinates the hypothetical answer?

Yes, that is the point. HyDE uses the generated answer only as a retrieval query, so factual errors in it do not reach the user. What matters is that the hypothetical answer uses answer-shaped vocabulary and structure, which lands closer to real answer chunks than the original question does. The retrieved chunks, not the hypothetical, are what the final generation is grounded in.

How much latency does query transformation add?

Each technique adds at least one extra model call before retrieval. A fast, small model keeps a rewrite or expansion in the low tens of milliseconds; multi-query and HyDE add more because they generate more or trigger several retrievals. Route conditionally so only the queries that need the heavier techniques pay for them, and keep the default path a single cheap rewrite.

Do I still need good chunking and embeddings if I transform queries?

Yes. Query transformation improves the query side of the match, but it cannot retrieve a chunk that was split badly or embedded with a weak model. Think of chunking, embeddings, query transformation, and reranking as four independent levers on the same retrieval quality number. Query transformation is usually the cheapest to add, which is why it is worth doing early, not a reason to skip the others.