Contextual retrieval: cut RAG failures without more infra

Your retriever returns the wrong chunks and you cannot figure out why. The chunk that holds the answer looks fine in isolation, the embedding model is fine, the reranker is fine. The problem is usually earlier: the chunk lost the context that made it findable the moment you split the document. Contextual retrieval fixes that at index time, and in Anthropic's tests it cut top-20 retrieval failures by up to 67 percent.

• Contextual retrieval prepends a short, chunk-specific context blurb to each chunk before you embed it and before you build the keyword index.
• On Anthropic's benchmark it moved the top-20 retrieval failure rate from 5.7 percent to 2.9 percent (a 49 percent drop), and to 1.9 percent (67 percent) once reranking is added.
• The one-time indexing cost is about 1.02 dollars per million document tokens when you use prompt caching.
• It earns its place once your corpus is too large to drop into a single prompt and your retriever misses on ambiguous, reference-heavy chunks.

Why a good chunk still fails to retrieve

Take a chunk that reads: "The company's revenue grew by 3 percent over the previous quarter." On its own it does not say which company, which quarter, or which year. When you embed that sentence, it lands in a generic neighborhood about revenue growth, nowhere near a query like "how did Acme perform in Q2 2025." The keyword index has the same problem: the token "Acme" is not in the chunk, so BM25 cannot match it either.

This is not an embedding-quality issue you can swap your way out of. The information needed to retrieve the chunk was in the surrounding pages, and chunking threw it away. The larger and more cross-referential your documents, the more often this happens: contracts that define a party once and then say "the Vendor" for 40 pages, support docs that set a product name in the title and never repeat it, financial filings full of "the period."

What contextual retrieval actually changes

Before you index, you run each chunk through a cheap language model along with the full source document and ask for a 50 to 100 token blurb that situates the chunk: which company, which section, which time period. You prepend that blurb to the chunk, then you build two indexes over the enriched text: a contextual embeddings index and a contextual keyword (BM25) index. At query time you search both and fuse the results, the same rank-fusion idea behind hybrid search with BM25 and embeddings.

Nothing about your query path gets slower or more expensive. The extra work is a one-time pass at ingestion, so the cost lands where you can batch and cache it rather than on every user request.

The numbers worth quoting to your team

Anthropic measured top-20 retrieval failure rate on a benchmark corpus:

• Baseline embeddings only: 5.7 percent failures.
• Contextual embeddings: 3.7 percent, a 35 percent reduction.
• Contextual embeddings plus contextual BM25: 2.9 percent, a 49 percent reduction.
• Add reranking of the top 150 candidates down to 20: 1.9 percent, a 67 percent reduction.

The jump from embeddings-only to embeddings-plus-BM25 is the cheapest win in that list. If you are still on pure vector search, adding a keyword index is the first thing to try, and contextual retrieval makes both indexes stronger at once.

What it costs to run

The obvious objection is that running every chunk through an LLM sounds expensive. With prompt caching it is not. You cache the full document once and vary only the chunk you are describing, which Anthropic reports cuts cost by up to 90 percent and latency by more than 2x. The net figure they publish is about 1.02 dollars per million document tokens, paid once at ingestion. For a 10 million token corpus that is roughly 10 dollars to reindex. If you are already using prompt caching to cut LLM cost elsewhere, this is the same lever applied to your pipeline.

Query-time cost does not change, because the enriched text lives in your index, not in the prompt you send per request.

When to reach for it, and when not to

Do not start here. Contextual retrieval is a multiplier on a sound pipeline, not a rescue for a broken one.

First, check whether you need retrieval at all. If your whole knowledge base fits in roughly 200,000 tokens, about 500 pages, you can put it directly in the prompt and skip the retrieval stack entirely. We walk through that tradeoff in long context versus RAG as a production decision.

Second, fix chunking before you enrich chunks. Bad boundaries that split a table or cut a sentence in half will still hurt after you add context. Get that right first, as covered in RAG chunking strategies for retrieval quality.

Third, treat reranking as a separate, stackable layer. It gave the largest single additional gain in the benchmark, and it composes cleanly with contextual retrieval. See two-stage retrieval with reranking for how to wire it in.

A sensible order of operations: solid chunking, then hybrid search, then contextual retrieval, then reranking. Measure after each step so you know which one actually moved your recall.

How to ship it without a reindex fire drill

Contextual retrieval means regenerating your index, so treat it like any embedding migration. Version your index, backfill the enriched embeddings in a shadow index, and cut over only after you compare recall on a held-out query set. The mechanics are the same ones we describe in upgrading an embedding model without breaking RAG, and they fold naturally into a production RAG architecture that already separates ingestion from serving.

If your retriever is guessing on ambiguous chunks, contextual retrieval is one of the highest-leverage changes available, and it does not add a single millisecond to your query path.

Frequently asked questions

Does contextual retrieval replace reranking?

No. They target different failures. Contextual retrieval makes each chunk more findable at index time; reranking reorders the candidates a retriever already surfaced. Anthropic's best result stacks both, so run them together rather than choosing one.

How long should the generated context be?

Fifty to one hundred tokens per chunk is the sweet spot. Long enough to name the entity, section, and time period the chunk assumes; short enough that it does not dilute the chunk's own signal.

Does it slow down queries?

No. The extra LLM work happens once at ingestion. At query time you are searching a normal index, so latency and per-request cost are unchanged.

Which model should generate the context?

A small, cheap model is fine for the context blurbs. Pair it with prompt caching over the full document so the marginal cost per chunk stays near a rounding error.