Search systems · Migration guide
Embedding Model Migration Without a Silent Search Regression

A new embedding model can improve retrieval quality, reduce latency, or simplify operations. Replacing it safely requires more than changing a model name: vectors must be versioned, evaluated on the same queries, and moved through a parallel index before production traffic depends on them.
The risk is compatibility, not just quality
In a retrieval-augmented generation (RAG) system, documents and user queries are converted into numerical vectors. A search service compares the query vector with stored document vectors and returns the nearest candidates to the language model.
When the embedding model changes, the new vectors are not automatically compatible with the old ones. Even if both models produce vectors with the same number of dimensions, their coordinate systems can represent meaning differently. A query encoded by the new model may therefore retrieve old vectors with an apparently valid score but an invalid semantic comparison.
This creates two dangerous failure modes:
- Obvious failure: the index rejects vectors because their dimensions do not match.
- Silent regression: the index accepts the vectors, but retrieval quality falls without an infrastructure error.
The safe default is to treat every model change as a schema and data migration. Create a new embedding version, rebuild a separate index, evaluate it against a fixed query set, and switch traffic only after the new path passes explicit checks.
A concrete case: changing the model behind a support assistant
Consider a support assistant that searches product documentation. Its current pipeline has these stages:
- Split source documents into chunks.
- Encode each chunk with the current model.
- Store vectors and metadata in a vector index.
- Encode each incoming question with the same model.
- Retrieve the top candidates and pass them to the generator.
The team wants to test a replacement model. The reason might be better multilingual retrieval, improved performance on short queries, lower memory use, or a new provider requirement. The reason is not important to the migration mechanics. The critical invariant is this:
A query and the documents it searches must be embedded with the same model version and compatible preprocessing.
During migration, the old path remains available while the new path is built and measured. Do not overwrite the old vectors in place.
1. Define the embedding contract before comparing models
Write down the properties that make one embedding deployment compatible with another. Store them as configuration rather than relying on an informal model name.
{
"embedding_version": "docs-2026-11-09-model-b-v1",
"model_id": "provider/model-b",
"dimensions": 1536,
"similarity": "cosine",
"document_prefix": "passage: ",
"query_prefix": "query: ",
"chunker_version": "markdown-chunker-v3",
"normalization": "l2",
"source_revision": "docs-2026-11-09"
}
The exact values above are illustrative configuration, not verified recommendations. Use the values required by the model and search engine you actually operate.
At minimum, record:
- Model identifier and provider.
- Model revision or deployment identifier, if available.
- Output dimensions and numeric type.
- Similarity metric, such as cosine similarity or dot product.
- Query and document instruction prefixes.
- Text normalization, truncation, and token limits.
- Chunker version and source-data revision.
- Embedding generation timestamp and software version.
A vector without this metadata is difficult to audit. Treat the embedding version as part of the index schema, not merely as a deployment label.
2. Build a small, representative query set
Compare models on queries that resemble real use. A benchmark made only from easy, hand-written examples can hide the failures that matter in production.
If you have historical search logs, sample queries after removing personal data and secrets. Otherwise, create a labeled set from the main user journeys. Include short and long queries, synonyms, product names, error messages, multilingual inputs if relevant, and queries whose correct answer depends on a specific version or condition.
Store each evaluation item in a reviewable format:
{
"id": "q-017",
"query": "How do I rotate an expired service token?",
"relevant_chunk_ids": [
"auth/token-rotation.md#section-3"
],
"acceptable_chunk_ids": [
"auth/token-rotation.md#section-2",
"auth/token-rotation.md#section-4"
],
"notes": "Must distinguish rotation from revocation."
}
The labels are the test oracle. They should describe what retrieval must return, not what a particular model happens to return. For a first comparison, a few dozen carefully reviewed queries can reveal obvious regressions. A larger set gives more confidence, but no query set proves universal quality.
3. Compare retrieval, not just vector statistics
Embedding dimensions, vector norms, and average similarity scores are useful diagnostics, but they are not a quality evaluation. A model can have plausible-looking distributions while returning the wrong chunks.
Run the same query set against each candidate index and calculate retrieval metrics such as:
- Recall@k: whether at least one relevant chunk appears in the first k results.
- MRR: how highly the first relevant result is ranked.
- nDCG: ranking quality when relevance has more than two levels.
- Zero-hit rate: how often no acceptable result appears in the inspected window.
- Latency: query encoding time plus index search time.
The metric choice should match the product. If the generator needs one authoritative chunk, rank position matters. If it can combine several sources, recall at a larger k may be more useful.
A minimal evaluation loop can look like this:
for query in evaluation_queries:
expected = query.relevant_chunk_ids + query.acceptable_chunk_ids
for index in ["current-v1", "candidate-v2"]:
vector = embed(
text=query.text,
embedding_version=index.embedding_version
)
results = index.search(vector, top_k=10)
record(
query_id=query.id,
index=index.name,
result_ids=[item.id for item in results],
latency_ms=results.latency_ms,
hit_at_10=any(item.id in expected for item in results)
)
This pseudocode is intentionally provider-neutral. Adapt the embedding and search calls to your stack. The important property is that each index receives a query encoded with its own embedding version.
4. Create a parallel index
The parallel index is the safety boundary. It contains a complete copy of the source corpus embedded with the candidate configuration. Give it a distinct name that includes the embedding version.
documents_current_v1
documents_candidate_v2
Keep the document identifier stable across both indexes. A stable ID makes result comparison and rollback easier, while the index name and embedding version identify the representation.
Re-embed every source chunk for the candidate index. Do not copy old vectors and update only recently changed documents. That produces a mixed corpus in which some candidates use the old representation and others use the new one.
If the source corpus changes during the rebuild, choose one of two explicit strategies:
- Capture a source revision, build from that snapshot, then replay changes before cutover.
- Continuously mirror changes into both indexes and verify that document counts and revisions converge before switching.
The first strategy is simpler to reason about. The second can reduce staleness for long rebuilds but requires more operational bookkeeping.
5. Use dual-read evaluation before dual-read production
Before changing user-visible results, send evaluation queries to both indexes. Compare not only aggregate metrics but also individual disagreements.
A useful review table contains:
| Query | Current top result | Candidate top result | Expected result | Decision |
|---|---|---|---|---|
| Expired token rotation | Revocation guide | Rotation guide | Rotation guide | Candidate improves |
| Legacy API timeout | Migration note | Generic timeout page | Migration note | Investigate |
The rows above are examples of the review format, not measured results. In a real migration, populate them from your query set and label every disagreement as an improvement, acceptable change, regression, or unresolved case.
Watch for systematic regressions: one language, one product area, short queries, numbers, code snippets, or documents with similar titles. Aggregate scores can conceal a concentrated failure that affects an important user journey.
6. Cut over with an explicit routing switch
The application should select an embedding version and index through configuration or a feature flag. Avoid hard-coding an index name in multiple services.
SEARCH_INDEX_CURRENT=documents_current_v1
SEARCH_INDEX_CANDIDATE=documents_candidate_v2
ACTIVE_SEARCH_INDEX=documents_current_v1
ACTIVE_EMBEDDING_VERSION=docs-2026-11-09-model-a-v1
A controlled cutover can proceed in stages:
- Build and validate the candidate index offline.
- Enable candidate searches for internal evaluation or a small, observable traffic slice.
- Compare latency, empty-result rate, retrieval metrics, and application-level answer signals.
- Increase traffic only when the candidate remains within agreed thresholds.
- Switch the default route to the candidate index.
- Keep the previous index available for rollback until the observation window ends.
The application must also switch query embedding at the same time as index routing. A candidate query vector sent to the current index is an incompatibility, even if the request succeeds.
Verification checklist
A migration is ready for cutover only when the following checks have evidence attached:
- The candidate index contains the intended source revision.
- Every candidate vector carries the expected embedding version.
- Dimensions and similarity metric match the candidate schema.
- Query encoding selects the same version as the target index.
- Evaluation queries have been run against both indexes.
- Relevant-hit metrics meet the agreed acceptance criteria.
- Representative disagreements have been reviewed.
- Search latency and resource use are within operational limits.
- Rollback changes one routing configuration, not a data rewrite.
- Monitoring can distinguish current and candidate traffic.
If a check is not measured, label it as unknown. Do not convert an unknown into an assumption because the index loaded successfully.
Failure cases to plan for
Mixed vectors in one index
This often happens when a partial backfill updates documents in place. The index may remain searchable while ranking becomes inconsistent. Use immutable index versions and rebuild the candidate corpus completely.
Same dimensions, different semantics
Matching dimensions do not establish compatibility. Store the model identifier and representation metadata with every index, and never infer compatibility from shape alone.
Different preprocessing
A new model may expect a query instruction, a document prefix, different truncation, or a different normalization policy. Put preprocessing in the embedding contract and test it with the same fixtures used in evaluation.
Stale candidate data
A candidate index built from yesterday’s corpus can appear better or worse for reasons unrelated to the model. Compare indexes built from the same source revision, or explicitly account for changes.
Overfitting to the evaluation set
A model can be tuned to a small hand-written benchmark and still fail on production traffic. Keep a held-out set, sample new anonymized queries after launch, and review failures continuously.
Rollback without query rollback
Switching only the document index back is unsafe if the query encoder remains on the candidate model. Rollback must restore the index and the matching query-embedding configuration together.
Limitations and practical trade-offs
Offline retrieval metrics do not measure the entire RAG system. A retrieved chunk can be relevant but poorly formatted for the generator. Conversely, a generator may answer correctly from a less highly ranked chunk. Add answer-quality checks only after retrieval itself is measurable.
A parallel rebuild temporarily costs more storage and compute. That cost buys a reversible migration boundary. If the corpus is very large, use a snapshot, batch embeddings, checkpoint progress, and preserve enough metadata to resume without silently mixing versions.
No single score decides whether a model is right. Consider retrieval quality, latency, memory, throughput, language coverage, data handling requirements, operational reliability, and the cost of maintaining the model. Make the decision from your query set and stated constraints rather than from dimension count or a generic benchmark alone.
Final operating pattern
The repeatable pattern is simple:
- Define an immutable embedding contract.
- Create a labeled query set representative of real search.
- Build a complete candidate index from the same source revision.
- Encode queries with the matching candidate version.
- Compare retrieval quality and operational behavior.
- Cut over through a routing switch with rollback ready.
- Keep monitoring after launch and review new failures.
This approach turns an opaque model replacement into a controlled data migration. The key protection against a silent search regression is not a larger model or a more impressive score. It is the combination of versioned representations, reproducible evaluation, and a parallel index that can be rejected without taking search offline.