Where the milliseconds go in your RAG pipeline

A RAG answer that feels slow is almost never slow everywhere. The time piles up in two or three predictable stages, and once you measure per stage the fix is usually obvious. The problem is that most teams ship a pipeline with no latency budget at all, then react to a p95 number they cannot explain.

A latency budget is simple: pick the p95 you owe the user, then hand each stage a slice of it. If a stage blows its slice, you know exactly where to look. This post gives a working budget for a typical retrieval-augmented pipeline and the moves that buy time back.

Start from a p95 target, then divide it

Do not start from "how fast is our vector database." Start from the number the product owes the user. For an interactive assistant, a p95 of 2 seconds to a useful first response is a reasonable bar; for a search-style feature where the user is scanning, 800ms to 1s is closer to the expectation.

Pick that number first, because it changes every downstream decision. A 2s budget lets you rerank and run a mid-sized model. An 800ms budget means you are probably skipping the reranker, caching aggressively, and streaming tokens so the first word lands early. If you have not decided whether to stream, read our note on streaming LLM responses in a SaaS feature before you set the budget, because streaming changes what "first response" even means.

Where the milliseconds actually go

Across production pipelines the distribution is lopsided. Generation dominates, retrieval-side work compounds when you add stages, and everything else is overhead you can usually squeeze. Here is the honest breakdown.

Query processing and rewriting

Embedding the query is cheap, often around 35ms. The expensive part is optional query rewriting or expansion, which quietly adds a whole model call. If you rewrite the query with an LLM before retrieval, you have doubled your model round-trips. That can be worth it for recall, but budget for it. Our write-up on query rewriting for retrieval covers when the extra call earns its place.

Retrieval and vector search

A well-indexed vector search over a few hundred thousand documents runs in tens of milliseconds: think 45ms p95, not 500ms. If your retrieval is taking hundreds of milliseconds, the cause is usually an unindexed filter, an oversized top-k, or a cold cache, not the database itself. Which store you picked matters less than how you configured it; our vector database comparison gets into the tradeoffs. Hybrid retrieval that runs keyword and vector search together adds a little fan-out but usually improves quality enough to justify it, as covered in hybrid search with BM25 and RRF.

Reranking

This is the stage teams underestimate. A cross-encoder reranker is the single largest latency contributor after generation, commonly adding 120ms p95 and sometimes far more if you rerank a large candidate set on CPU. Reranking almost always helps quality. It is also the first thing to drop when your budget is tight. Rerank fewer candidates, use a smaller reranker, or move it to a GPU before you decide it is non-negotiable.

Generation

Generation is where the budget lives or dies. In most real pipelines it is 60 to 85 percent of total latency. A tight case study that took p95 from 2000ms to 800ms ended up with generation at roughly 650ms, about 80 percent of the remaining time, and everything else was rounding error. The two levers are time to first token and total tokens produced. If you stream, the user feels time to first token, not total time, which is why time to first token is the metric to optimize for perceived speed.

A worked budget for a 2-second p95

Here is a budget you can copy and then adjust against your own traces. It assumes a streaming interactive assistant with a 2s p95 to a useful response.

• Query embedding and rewriting: 150ms. Skip the rewrite if you are over budget.
• Retrieval, including hybrid fan-out: 100ms. If this is higher, fix the index before anything else.
• Reranking: 150ms on a small reranker over a trimmed candidate set.
• Prompt assembly and network overhead: 100ms. Real, and easy to forget.
• Generation to first token: 400ms. This is what the user perceives when streaming.
• Headroom for retries, cold caches, and provider variance: about 1s.

The headroom line is not padding. Provider latency has a fat tail, and a single retry can double your worst case. Budget for the bad path, not the demo.

Five moves that buy back time

When a stage overruns, reach for these in order of return on effort.

1. Cache. A semantic cache serves repeat and near-repeat questions in single-digit milliseconds and removes the whole generation cost for a real slice of traffic. See semantic caching for LLM cost.
2. Trim the candidate set before reranking. Rerank 20 candidates, not 200.
3. Shrink the prompt. Fewer retrieved chunks and tighter chunking strategies cut both generation tokens and reranking work.
4. Right-size the model. A smaller or lower-effort model on the easy majority of queries pays off; our note on reasoning effort versus cost and latency shows the tradeoff.
5. Parallelize. Run retrieval and any metadata lookups concurrently instead of in series.

What to measure, per stage

You cannot budget what you do not trace. Instrument each stage with its own timer and record p50, p95, and p99 separately, because averages hide the tail that actually pages you. Tag spans with model, top-k, and cache hit so you can slice latency by request shape. A p95 that is fine overall but terrible for long prompts is a signal you would miss on the average. Wiring this into your existing observability is part of how you monitor an LLM feature in production.

Frequently asked questions

What is a good p95 latency for a RAG feature?

It depends on the interaction. For a conversational assistant that streams, a 2s p95 to a useful first response is reasonable and 400 to 600ms time to first token feels responsive. For a search-style result the user scans, aim under 1s. Set the number from the product experience, not from what your stack happens to do.

Which RAG stage should I optimize first?

Trace first, then optimize the biggest slice. In most pipelines that is generation, so caching and model right-sizing win first. If your retrieval is in the hundreds of milliseconds, fix the index before touching anything else, because that is a configuration bug, not a fundamental cost.

Does reranking always slow things down too much to keep?

No. Reranking usually improves answer quality enough to keep. The trick is to rerank a small candidate set on adequate hardware. Rerank 20 candidates with a small cross-encoder rather than 200 on CPU, and its cost stays inside a 150ms slice.

Can streaming fix a slow pipeline?

Streaming fixes perceived latency, not real latency. It lets the first token arrive early so the user starts reading while the rest generates. It does nothing for total cost or for non-streaming consumers like an API that needs the full answer, so treat it as a perception tool on top of a real budget, not a substitute for one.

Set the budget, trace every stage, and the pipeline stops being a mystery. If you want a second pair of eyes on a RAG feature that is slower than it should be, that is the kind of task Boundev ships in a week.