What is query rewriting?
Query rewriting is the step that turns a user's raw question into one or more search queries better suited to the retriever, by rephrasing, expanding, splitting or adding context from the conversation, before retrieval runs.
Checked
People ask questions conversationally: "what about the second one?", "is that still true after the update?". A retriever sees only the words in front of it. Without the earlier turns, those questions match nothing useful.
Query rewriting puts a model between the user and the index. It resolves references, removes filler, splits compound questions and sometimes writes a guess at the answer to search with. In agentic systems it stops being a single step and becomes part of the loop.
Common techniques
Condense and rephrase
Rewrite a follow-up into a standalone query using the conversation. OpenAI's vector store search offers this as rewrite_query, turning "I'd like to know the height of the main office building." into "primary office building height".
Decompose
Split a multi-part question into separate searches and retrieve for each, so one part does not crowd out the other.
HyDE
Have a model write a hypothetical answer document and search with its embedding. Introduced by Luyu Gao, Xueguang Ma, Jimmy Lin and Jamie Callan on 2022-12-20; the encoder is meant to keep the relevance pattern and filter out invented details.
Trained rewriters
Rewrite-Retrieve-Read (Ma et al., 2023-05-23) trains a small rewriter with reinforcement learning from the answering model's feedback, instead of prompting a large model each time.
Query rewriting inside agents
When retrieval is a tool the agent calls, the agent rewrites its own queries as it learns what is in the corpus. Anthropic's write-up of its multi-agent research system (2025-06-13) found agents defaulted to overly long, specific queries that returned few results. Anthropic's fix was to prompt them to start with short, broad queries, narrow down, and refine the next query after each result.
Where query rewriting fails
Intent drift
A rewrite can quietly change the question, for example dropping a negation or a date range, and retrieval then answers a question nobody asked.
Invented specifics
HyDE-style rewrites can add plausible but wrong names or codes. For exact-identifier lookups, the original query should still go to keyword search.
Latency and cost
Each rewrite is an extra model call before retrieval starts. A small, fast model usually does the job.
Invisible failures
If the rewritten query is not logged, retrieval bugs look like model bugs. OpenAI returns the rewritten query in a search_query field for this reason.
How Motiontic's engineers apply it
Motiontic's engineers add query rewriting when the eval set shows conversational follow-ups or compound questions failing retrieval, and log the original and rewritten queries in every trace. The original query is usually kept as one of the searches, so a bad rewrite cannot remove the plain-text match.
Query rewriting techniques compared
| Condense and rephrase | Decompose | HyDE | |
|---|---|---|---|
| Changes | Wording, references | Number of queries | Searches with a generated answer |
| Helps with | Follow-up questions | Multi-part questions | Short or vague questions |
| Extra cost | One model call | One call plus several searches | One generation plus embedding |
| Main risk | Intent drift | Lost connections between parts | Invented details |
Common questions
What is HyDE in RAG?
Hypothetical Document Embeddings: a model writes a fake answer to the question, and the system searches with that answer's embedding, which often sits closer to real answer passages than the short question does.
Does query rewriting help hybrid search?
Often. A rewrite can produce clean keywords for BM25 and a natural-language query for vectors. Keeping the original query in the mix protects exact terms.
Does OpenAI's file search rewrite queries?
Vector store search accepts rewrite_query=true, which rewrites the query automatically and returns the rewritten version in the search_query field.
What is the difference between query rewriting and agentic search?
Query rewriting is one transformation before retrieval. In agentic search the agent issues, reads and reformulates queries in a loop until it has enough evidence.
Related
Agentic search
Retrieval as a tool the agent calls, step by step, rather than a pipeline stage.
Hybrid search
Keyword ranking and vector similarity, fused into one result list.
Retrieval-augmented generation (RAG)
Fetch the right passages at query time, then answer from them with citations.
RAG vs Agentic Search
One retrieval before the answer, compared with an agent that searches as it works.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Sources
- Gao, Ma, Lin and Callan, Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE, arXiv, 2022-12-20)
- Ma et al., Query Rewriting for Retrieval-Augmented Large Language Models (arXiv, 2023-05-23)
- OpenAI, Retrieval guide (vector store search, chunking, hybrid ranking)
- Anthropic, How the multi-agent research system was built (2025-06-13)
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.