RAG reranking explained: why vector similarity fails and how cross-encoders fix it

Vector embeddings are a remarkable engineering shortcut. By converting sentences and paragraphs into floating-point vectors, systems can search through millions of documents in sub-10 millisecond latencies using approximate nearest neighbor (ANN) indexes.

However, compressing an entire 500-word passage into a single geometric vector comes with a severe compression penalty. A bi-encoder embeds the query and the document completely independently without ever allowing their tokens to interact. Consequently, vector search struggles with negation ("looking for Python, NOT Django"), precise numeric ranges, and domain-specific abbreviations.

This is where reranking in RAG transforms performance. Instead of feeding the top-5 raw vector matches directly to the generative LLM, production pipelines employ a two-stage retrieval architecture.

• Bi-encoders process query and document independently, losing fine token interactions.
• Negative filters and numeric conditions are frequently distorted by dense embeddings.
• Two-stage retrieval repairs these blind spots by introducing deep cross-attention.

To understand why rerankers outperform raw vector search, developers must understand the fundamental difference between Bi-Encoders and Cross-Encoders.

A two-stage retrieval pipeline combines the complementary strengths of bi-encoders (speed and scale) and cross-encoders (deep semantic precision).

In Stage 1 (Candidate Generation), a hybrid search engine retrieves a wide candidate pool—typically between 50 and 100 documents—using a mix of vector similarity and BM25 keyword matching. This stage prioritizes high recall over precision.

In Stage 2 (Precision Scoring), cross-encoder reranking scores every candidate against the original user query, computing a relevance score. The pipeline sorts candidates by this score, strips away low-scoring distractors, and sends only the top 3 to 5 pristine chunks to the LLM prompt.

For high-throughput systems where full cross-encoder scoring is too computationally expensive, ColBERT (Contextualized Late Interaction over BERT) offers an elegant intermediate solution.

ColBERT computes token-level vector representations for both the query and document, storing multiple vectors per chunk. At search time, it computes the maximum similarity (MaxSim) across token pairs.

This allows ColBERT to retain token-level cross-attention expressiveness while operating orders of magnitude faster than traditional cross-encoders, making it ideal for sub-50ms enterprise retrieval workloads.

• Late interaction preserves token-level contextual representations without full cross-transformer cost.
• MaxSim operators rapidly aggregate pairwise token matches across indexed document passages.
• Delivers 95% of cross-encoder accuracy at a fraction of the computational overhead.

Developers often ask: is reranking worth the additional latency? In production benchmarks across enterprise knowledge bases, passing 50 candidates through a modern reranker adds between 35ms and 65ms of latency.

However, because rerankers eliminate irrelevant context, you can safely decrease the final context window size from 10 chunks down to 3 chunks. This saves 2,000+ input tokens per prompt, accelerating LLM generation time and reducing overall end-to-end latency.