RAG diagnostics · Advanced · 15 min read

Diagnosing a RAG Index with Embedding Projection Maps

Диагностируем RAG-индекс по картам векторных представлений
Temporary fallback cover; replace in editorial pass.

A reproducible method for testing whether a projection reveals chunking and indexing defects

A projection map can make a retrieval system look understandable without proving that it retrieves the right evidence. The useful question is narrower: can the map help you find suspicious neighborhoods that you then verify against the original vectors, metadata, and source text? This guide presents that workflow.

In a retrieval-augmented generation (RAG) system, a query is embedded, compared with indexed document chunks, and passed to a language model as context. When answers become inconsistent, the defect may be in the embedding model, the chunking policy, metadata filters, vector normalization, or the approximate nearest-neighbor index. A two-dimensional picture can suggest where to look, but it cannot replace retrieval evaluation.

What the map can and cannot tell you

A projection such as UMAP or t-SNE maps high-dimensional embedding vectors into two dimensions. It preserves some relationships, but not all of them. Distances, cluster sizes, empty regions, and apparent boundaries can change during projection.

The map is useful for asking questions such as:

It is not sufficient evidence for claims such as “the index is correct,” “this cluster is a topic,” or “the model understands document structure.” Those claims require measurements in the original vector space and, where possible, labeled retrieval tests.

A concrete diagnostic case

Consider an internal policy collection. Each source file contains headings, paragraphs, lists, and tables. A retrieval failure is reported as follows: a question about a password-reset exception retrieves the general password policy, but not the short exception section that follows it.

There are several plausible causes. The exception may have been split from its heading; a table may have been flattened into unrelated rows; the chunk may have been indexed with the wrong tenant or document identifier; or the approximate index may be returning a different neighbor from the exact cosine search.

The diagnostic dataset should therefore retain, for every vector, the chunk text or a secure reference to it, the source document identifier, the page or section identifier, the character offsets, the embedding-model identifier, the chunking configuration, and the index-build identifier. Without those fields, a visual anomaly is difficult to trace back to an actionable defect.

Step 1: Freeze the experiment inputs

Do not begin by exporting an arbitrary sample from a live index. Create a reproducible snapshot or manifest. Record the exact vector IDs, vector dimension, distance metric, normalization rule, embedding model and version, chunking configuration, metadata-filter policy, and index parameters.

A minimal manifest can look like this:

{
  "snapshot": "2026-11-17T12:00:00Z",
  "embedding_model": "example-model-id",
  "dimension": 1536,
  "metric": "cosine",
  "normalized": true,
  "chunking": {
    "strategy": "heading_aware",
    "target_tokens": 450,
    "overlap_tokens": 60
  },
  "index": {
    "type": "example-ann-index",
    "build_id": "example-build-id"
  }
}

The identifiers above are placeholders, not observed production values. Replace them with values from the system under investigation. Keep credentials and raw personal data out of the diagnostic export.

Step 2: Build a projection with traceable points

Sample vectors deterministically or project the complete bounded snapshot. A deterministic sample makes reruns comparable. Color points by a field that is independent of the projection, such as source document or section, and use shape or a second view for chunk position. Avoid coloring only by an inferred topic: that can make the desired pattern appear by construction.

The following Python sketch shows the data flow. It is an example, not a claim that it was executed here:

import json
import numpy as np
import pandas as pd
from sklearn.manifold import TSNE

# vectors.npy: shape (n_items, dimension)
# chunks.parquet: id, document_id, section_id, text, offset_start, offset_end
vectors = np.load("vectors.npy")
chunks = pd.read_parquet("chunks.parquet")

assert len(vectors) == len(chunks)
assert np.isfinite(vectors).all()

# For cosine diagnostics, normalize explicitly unless the export is known
# to contain unit-length vectors.
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
unit_vectors = vectors / np.maximum(norms, 1e-12)

rng = np.random.default_rng(41)
sample_size = min(10000, len(unit_vectors))
sample_idx = np.sort(rng.choice(len(unit_vectors), sample_size, replace=False))

projected = TSNE(
    n_components=2,
    metric="cosine",
    init="random",
    random_state=41,
    perplexity=30
).fit_transform(unit_vectors[sample_idx])

out = chunks.iloc[sample_idx].copy()
out["x"] = projected[:, 0]
out["y"] = projected[:, 1]
out.to_parquet("projection.parquet", index=False)

For large collections, fit the projection on a fixed sample and then transform only if the chosen method supports a reliable transform operation. If it does not, state clearly that separate runs are not directly comparable. Change one parameter at a time and record it with the output.

Step 3: Measure retained neighbors

A projection is most useful when you quantify how much local structure it retains. For each sampled point, compute its exact top-k neighbors in the original vector space. Then compute its top-k neighbors in the two-dimensional projection. The retained-neighbor rate is the intersection size divided by k.

For a point i, let Nhigh(i) be its exact high-dimensional neighbors and Nmap(i) be its projected-space neighbors. The per-point score is:

retained_neighbors(i, k) =
    |N_high(i) ∩ N_map(i)| / k

Report the median, lower quantiles, and distribution of this score, not only a single average. A map with a moderate global score may still be useful for locating dense suspicious regions, while a low score means that visual proximity should be treated very cautiously.

Example diagnostic code for a bounded sample:

from sklearn.metrics import pairwise_distances
from sklearn.neighbors import NearestNeighbors

k = 10

high_nn = NearestNeighbors(
    n_neighbors=k + 1,
    metric="cosine"
).fit(unit_vectors[sample_idx])

_, high_ids = high_nn.kneighbors(unit_vectors[sample_idx])
high_ids = high_ids[:, 1:]  # remove self

map_nn = NearestNeighbors(
    n_neighbors=k + 1,
    metric="euclidean"
).fit(projected)

_, map_ids = map_nn.kneighbors(projected)
map_ids = map_ids[:, 1:]  # remove self

retained = np.array([
    len(set(a).intersection(b)) / k
    for a, b in zip(high_ids, map_ids)
])

summary = {
    "n_points": int(len(retained)),
    "k": k,
    "median": float(np.median(retained)),
    "p10": float(np.quantile(retained, 0.10)),
    "p90": float(np.quantile(retained, 0.90))
}
print(summary)

This measures agreement between two neighbor graphs on the same sample. It does not measure retrieval quality for user queries. To measure that, use a separate query set with expected documents or passages.

Step 4: Check exact cosine neighbors

Approximate nearest-neighbor search is an optimization. It may return a neighbor set that differs from an exhaustive search, especially when vectors are close in similarity or when search parameters are intentionally low for latency. Verify suspicious points against exact cosine neighbors.

If vectors are unit-normalized, cosine similarity is the dot product. For a query vector q and chunk vector x:

cosine(q, x) = q · x

The exact reference implementation below is suitable for a bounded diagnostic batch. It is not intended to replace a production index:

def exact_cosine_top_k(query_vectors, document_vectors, k):
    # Both inputs should be normalized before this function.
    scores = query_vectors @ document_vectors.T
    top = np.argpartition(-scores, kth=k - 1, axis=1)[:, :k]

    rows = []
    for row, candidate_ids in enumerate(top):
        ordered = candidate_ids[
            np.argsort(-scores[row, candidate_ids])
        ]
        rows.append([
            (int(idx), float(scores[row, idx]))
            for idx in ordered
        ])
    return rows

# Example use:
# exact = exact_cosine_top_k(unit_queries, unit_vectors, k=10)

Compare the exact result with the result returned by the deployed index for the same vector, filters, and k. Record overlap, rank changes, and similarity margins. A mismatch is a lead, not automatically a bug. Check index configuration, deleted-vector handling, filter semantics, quantization, and whether the two paths use the same normalization.

Step 5: Trace suspicious neighborhoods to chunk boundaries

The most actionable use of a projection is often not “finding topics.” It is finding local groups whose source boundaries look wrong.

For each suspicious neighborhood:

  1. List the vector IDs and exact cosine similarities.
  2. Join them to document, section, page, and character-offset metadata.
  3. Sort the source records by document and source offset.
  4. Read the preceding and following chunks, not only the retrieved text.
  5. Check whether headings, definitions, table headers, list labels, and exceptions were separated from the content they govern.
  6. Re-embed a small corrected set and compare the exact neighbor lists.

Common boundary defects include a heading placed in the previous chunk, a table split between rows and its header, a definition separated from its term, a sentence cut between a condition and its exception, and overlap that copies boilerplate while omitting the decisive sentence. These are source-structure defects; a map can help locate them, but only the source text confirms them.

Useful indicators

Indicator What to calculate What it can suggest
Neighbor retention Intersection of exact and projected top-k neighbors Whether local visual proximity is a reasonable inspection aid
Exact/ANN overlap Intersection of exhaustive and deployed-index results Possible index, metric, filter, or normalization mismatch
Duplicate rate Repeated hashes or near-identical vectors Repeated ingestion, empty chunks, or parser failures
Boundary continuity Neighbor metadata versus source offsets and section IDs Chunks that split semantic units or lose structural context
Query recall Expected passage presence in top-k results Whether the index serves the task, independently of the map

Verification protocol

A practical verification run should produce an artifact containing the manifest, sampled IDs, projection parameters, retained-neighbor summary, exact-versus-ANN comparison, and a review table for suspicious neighborhoods. The review table should include a verdict such as “confirmed chunk boundary defect,” “index mismatch,” “projection artifact,” or “inconclusive,” with a short reason.

Re-run the same snapshot with the same random seed. The sampled IDs and summary should remain stable. Then change one controlled variable: for example, the projection seed, the sample size, the ANN search parameter, or the chunking configuration. A conclusion that disappears under a harmless projection change is weak evidence; a source-level defect that persists across exact-neighbor checks is stronger evidence.

For query evaluation, keep a held-out set of questions and expected evidence. Report recall at the chosen k, rank distribution, and failure examples. Do not infer query recall from cluster appearance or neighbor-retention scores.

Failure cases and limitations

What a defensible conclusion looks like

A defensible conclusion connects three layers of evidence: the projection identifies a candidate neighborhood; exact cosine search confirms or rejects the original-space relationship; and source inspection explains the behavior in terms of chunk boundaries, metadata, or index configuration.

For example, it is reasonable to write: “In the fixed snapshot, several exception chunks were isolated from their headings. Their exact cosine neighbors were dominated by generic policy chunks, and re-chunking the source kept the heading with the exception in a bounded test.” It is not reasonable to write: “The map proves the embedding model is confused.”

That distinction is the point of the method. Embedding projection maps are valuable as an inspection surface, but the diagnosis comes from reproducible measurements and traceable source evidence.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →