How to upgrade your RAG embedding model without breaking search

Upgrading your RAG embedding model is a data migration, not a settings change. Every vector in your index was produced by the old model and lives in that model's coordinate space, so queries embedded by a new model cannot be compared against old vectors and still return sensible neighbors. The safe path is to re-embed your whole corpus into a second index in the background, gate the switch on a retrieval eval, then ramp traffic behind a flag with instant rollback. This post walks through that zero-downtime pattern, the costs teams forget, and when a model upgrade is not worth doing at all.

Why swapping an embedding model is a corpus migration

A vector index is not neutral storage. Each document embedding is a point that only means something relative to the exact model that produced it. Two embedding models can both output 1024-dimensional vectors and still place the same sentence in completely different regions, because they learned different geometry. There is no shared frame of reference between them.

This is why the tempting one-line change - point the embedding call at the newer model and redeploy - quietly corrupts retrieval. Your stored document vectors still come from the old model. Your new queries come from the new model. Cosine similarity between them is close to noise. The system does not error. It returns results that look plausible and are subtly wrong, which is the worst failure mode a search feature can have.

So the real unit of work is not the model reference in your config. It is the corpus. Upgrading the model means re-embedding every chunk you have ever indexed, and doing it in a way that never leaves queries pointed at a half-converted index. If you have read our production RAG architecture guide, this is the operational sequel: the day you outgrow your first embedding model.

What actually breaks when you mix embedding spaces

It helps to be concrete about the failure, because it does not look like a failure. Three things go wrong at once, and none of them throws an exception.

Recall collapses silently. Queries stop finding the documents that answer them, because the query vector and the answer vector are now measured in incompatible spaces. Your top-k results fill up with whatever happens to be nearest in the mismatched geometry, which is often unrelated.

Relevance scores stay high. Similarity numbers do not drop to zero when spaces are mismatched; they cluster in a misleading middle range. Any threshold you tuned on the old model (drop results below 0.75, for example) is now meaningless, so a confidence cutoff will not save you.

The model downstream improvises. In a RAG pipeline the retrieved chunks feed a generation step. When retrieval quietly degrades, the language model does not announce that its context is weak. It answers anyway, grounded in the wrong passages. This is the same class of problem we covered in reducing hallucinations in production RAG: bad retrieval is invisible until a user reports a confidently wrong answer.

The zero-downtime re-embedding pattern

The pattern that works treats the upgrade like any risky production change: build the new thing beside the old thing, prove it is better, then shift traffic gradually with a way back. Four steps.

1. Add a second index, do not mutate the first

Create a parallel vector column or a second collection for the new model, sized for its dimension count. Keep the old index serving every live query untouched. Depending on which vector database you picked, this is a new pgvector column, a new Qdrant collection, or a second Pinecone namespace. The point is that reads never depend on a partially written index. At no moment is a query answered from a mix of old and new vectors.

2. Backfill embeddings in the background

Re-embed the corpus as a batched background job, writing new vectors into the second index while the old one serves production. Use a durable cursor so the job can resume after a crash, batch the embedding API calls to stay under rate limits, and record progress so you know exactly how much of the corpus is converted. For a large corpus this runs for hours or a weekend, and that is fine, because nothing user-facing depends on it finishing quickly. Re-chunk here if you also want to change chunk size, since the same job is already touching every document - see our notes on chunking strategies for retrieval quality.

3. Gate the cutover on a retrieval eval, not a hunch

Before any traffic moves, run the same fixed query set against both indexes and compare. This is where a real retrieval eval set earns its keep: a few dozen to a few hundred labeled query-to-document pairs drawn from actual usage, scored on recall at k and answer correctness. A model that wins on a public benchmark can lose on your corpus, because your documents and your users' phrasing are not the benchmark. Treat the new model as guilty until the eval says it beats the old one on your data. This is the same eval-gated discipline you should already apply to provider model updates.

4. Ramp behind a flag, keep instant rollback

Once the eval passes, route a small slice of live queries to the new index behind a feature flag, watch retrieval metrics and user signals, then increase the percentage. Because both indexes are live, rollback is a flag toggle, not a redeploy or a restore. This is the same shadow-then-ramp discipline in shipping AI features without breaking production. Only after the new index has served full traffic cleanly for a while do you drop the old column and reclaim the storage.

A worked example

Consider a B2B support product with a knowledge-base search over roughly 400,000 chunks, embedded a year ago with a then-current model. A newer embedding model promises better recall on multi-part questions. The numbers below are illustrative, not measured, but the shape is what teams actually hit.

The naive path: change the model reference, redeploy, done in an afternoon. Within a day, support agents notice the assistant "got worse" - it stops surfacing the right policy article for questions like "why was I charged twice after downgrading." Nobody sees an error. The similarity scores still read 0.6 to 0.8. It takes six days and an angry customer thread to trace it back to the model swap, because retrieval degraded without any alarm.

The migration path: add a second pgvector column for the new model, backfill 400,000 chunks over a weekend for a few dollars of embedding cost, then run 200 labeled support queries against both. The eval shows recall at 5 rising from 0.71 to 0.83, so the upgrade is real. Ramp from 5 percent to 100 percent of queries over three days, watching the deflection rate hold, then drop the old column. Same destination, but the failure is caught in the eval instead of in a customer's inbox.

The costs teams forget

The embedding API bill for re-embedding is usually the smallest line. The ones that surprise people:

Storage doubles during the migration. Two full sets of vectors coexist until you drop the old one, so plan index capacity for roughly double the corpus for the duration. On a managed vector service that is a real, if temporary, cost.

Dimension changes ripple. If the new model outputs a different vector width, the new column, index parameters, and any distance-metric assumptions all change with it. This is not a drop-in swap even at the schema level.

Hybrid search needs re-tuning. If you blend keyword and vector scores, the fusion weights were tuned for the old vectors. After the swap you have to re-tune them, as covered in hybrid search that blends BM25 and embeddings. Skipping this quietly undoes half the gain.

Backfill competes for rate limits. A large background embedding job and your live traffic pull from the same provider quota. Without separate limits or a throttle, the migration can starve production calls.

When not to upgrade at all

A newer model is not a reason on its own. Re-embedding is real operational work with a real blast radius, so it needs a payoff you can measure on your own eval. Skip or defer the upgrade when the new model wins on a public leaderboard but shows no gain on your labeled query set, when your retrieval quality is already above where users feel friction, or when the same effort spent on chunking, reranking, or hybrid search would move recall more than a model swap would.

The honest default is to leave a working embedding model in place and revisit only when your eval shows retrieval is the bottleneck. A model upgrade is worth doing when the eval proves it, and worth ignoring when it does not. For the wider decision framework, our AI engineering playbook covers where retrieval quality sits relative to the other levers.

Frequently asked questions

Can I store old and new embeddings in the same index and query both?

No. Vectors from different models occupy different coordinate spaces, so a single nearest-neighbor query cannot compare across them meaningfully. Keep them in separate indexes and query one model's vectors with that same model's query embeddings.

Do I have to re-embed everything, or can I convert vectors?

You have to re-embed. There is no reliable transform that maps one model's vectors into another model's space, so every chunk must be run through the new model to get a comparable vector.

How do I know the new model is actually better for my use case?

Run a fixed, labeled query set from real usage against both indexes and compare recall at k and answer correctness. Public benchmark wins do not transfer to your corpus. If the new model does not beat the old one on your data, the upgrade is not worth the migration.

How long should I keep the old index after cutover?

Keep it until the new index has served full production traffic cleanly long enough to trust the metrics, then drop it to reclaim storage. While both exist, rollback is a flag toggle, which is the main reason to hold the old index a little longer than feels necessary.