Advanced RAG retrieval techniques: moving beyond basic chunking

Every developer who builds a basic RAG pipeline runs into an unavoidable contradiction: what size should your chunks be?

If you make chunks small (e.g. 100 tokens), your embedding model generates laser-focused vectors that match user queries with high precision. However, when those small chunks are passed to the generator LLM, the model lacks the surrounding context needed to understand the broader nuance, often missing key qualifiers located in neighboring sentences.

If you make chunks large (e.g. 1,000 tokens), the chunk contains rich narrative context, but the embedding model averages the meaning across too many disparate sentences. The resulting vector becomes diluted, lowering cosine similarity and causing search to miss the chunk entirely.

Advanced RAG techniques solve this dilemma by decoupling the text that is searched from the text that is fed to the model.

The two most powerful patterns for resolving the chunk-size dilemma are Sentence-Window Retrieval and Parent-Document Retrieval.

In Sentence-Window Retrieval, text is split into individual sentences, and each sentence is embedded independently with its exact position tracked. When a user query matches a sentence vector, the retrieval engine does not return just that isolated sentence; it fetches a sliding window of surrounding sentences (e.g. 3 sentences before and 3 sentences after). The search achieves high embedding precision, while the LLM receives complete narrative context.

Parent-Document Retrieval expands this concept to hierarchical document structures. Documents are parsed into large parent sections (e.g. 1,000 tokens) and then subdivided into small child chunks (e.g. 150 tokens). The vector database indexes only the child chunks. When a child chunk matches the query, the system swaps it out and returns the entire parent section to the LLM, preserving complete tables, code blocks, and section continuity.

Retrieval failures often begin with the user’s query. Real users ask vague, misspelled, or slang-filled questions that do not resemble formal documentation language.

Query Expansion uses an LLM to generate 3 to 5 related search phrases or sub-queries. The system executes these queries in parallel, combining and deduplicating the retrieved chunks. This dramatically boosts recall for complex domain questions.

Hypothetical Document Embeddings (HyDE) takes a radical approach: instead of embedding the user’s question (which may look linguistically very different from the answer), it asks an LLM to generate a speculative, hypothetical answer. Even if the hypothetical answer contains minor factual errors, its linguistic style, terminology, and semantic structure closely mirror the true source documentation. Embedding the hypothetical answer retrieves real documents with startling accuracy.

The final line of defense against hallucinations is post-retrieval reranking. Standard vector databases use bi-encoder embeddings, which evaluate the query and documents separately. While bi-encoders can search across millions of documents in milliseconds, they lack deep cross-attentional nuance.

A cross-encoder reranker (such as Cohere Rerank or BGE-Reranker) analyzes the query and candidate chunk together, computing full cross-attention between every token in the query and every token in the candidate passage.

By pulling the top 30 results from vector search and filtering them down to the top 4 using a cross-encoder, you eliminate up to 70% of false-positive chunks, providing pristine grounding for the final generative model.

• Adopt sentence-window retrieval for narrative text where surrounding context is essential
• Use parent-document retrieval when working with structured technical manuals or legal contracts
• Implement query rewriting to turn vague conversational prompts into crisp search terms
• Always place a cross-encoder reranker between your vector database and your generator prompt