What is contextual retrieval?
Contextual retrieval is a technique, published by Anthropic in September 2024, that prepends a short model-written explanation of each chunk's place in its source document before the chunk is embedded and indexed for keyword search.
Checked
Retrieval systems split documents into chunks, and a chunk on its own often loses the facts that made it findable. A line that says the limit was raised to 50 units does not say which product, which customer tier or which year.
Contextual retrieval fixes this at indexing time. A model reads the whole document and each chunk, writes a short note that situates the chunk, and that note is stored with the chunk in both the vector index and the BM25 index.
How it works
Generate the context
In Anthropic's setup, Claude 3 Haiku received the full document and one chunk, and wrote a context that is usually 50 to 100 tokens long.
Prepend and index
The context is added in front of the chunk before embedding and before building the BM25 index, so both searches see it.
Keep cost down with caching
The document is cached once and reused for every chunk. Anthropic estimated a one-time cost of about $1.02 per million document tokens with prompt caching.
What Anthropic measured
Across several domains, with Gemini Text 004 embeddings, the top-20 retrieval failure rate fell from 5.7% to 3.7% with contextual embeddings, to 2.9% when contextual BM25 was added, and to 1.9% when a reranker was added on top.
The same post notes a simpler option: a knowledge base under about 200,000 tokens, roughly 500 pages, can go straight into the prompt with prompt caching and skip retrieval altogether.
How Motiontic's engineers apply it
Motiontic's engineers try contextual retrieval when evals show chunks failing because they lack names, dates or section headings that only appear elsewhere in the document. The context prompt is tuned per document type, such as contracts, tickets or manuals, and re-indexing cost is estimated before the corpus is reprocessed.
Anthropic's Contextual Retrieval results (top-20 retrieval failure rate)
| Configuration | Failure rate | Reduction |
|---|---|---|
| Baseline embeddings | 5.7% | None |
| Contextual embeddings | 3.7% | 35% |
| Contextual embeddings and contextual BM25 | 2.9% | 49% |
| Both, plus a reranker | 1.9% | 67% |
Common questions
What is the difference between contextual retrieval and standard RAG?
Standard RAG indexes chunks as they are. Contextual retrieval first adds a short, model-written description of where each chunk sits in its document, so the chunk can be found by queries that mention context it does not contain.
How much does contextual retrieval cost?
Anthropic estimated about $1.02 per million document tokens as a one-time indexing cost, using prompt caching so the full document is not paid for again for every chunk.
When is retrieval not needed at all?
Anthropic's post notes that a knowledge base under about 200,000 tokens can be placed directly in the prompt, with prompt caching to cut cost and latency.
Does contextual retrieval work with any embedding model?
The method changes the text that gets embedded, not the model, so it works with any embedding model and any BM25 index. Results vary by corpus and should be checked on an eval set.
Related
Hybrid search
Keyword ranking and vector similarity, fused into one result list.
Reranker
A second pass that rereads the shortlist and puts the right passages first.
Prompt caching
Paying less for the part of the prompt that does not change between calls.
RAG vs Fine-Tuning
Fetching the right facts at query time, compared with retraining the model's behaviour.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.