The 8 most common RAG mistakes developers make in production

Almost every RAG system begins as an exciting notebook demo: you load five PDFs, run an out-of-the-box chunker, query an embedding API, and ask three carefully chosen questions. The answers look brilliant, and the project is approved for production.

Two weeks after deployment, the complaints roll in: "The bot made up a refund policy that doesn’t exist", "It can’t find the documentation for error code 403", "It contradicts itself when I ask follow-up questions."

RAG failures are rarely caused by the underlying foundation model. They are caused by subtle architectural anti-patterns in data preparation, search design, and prompt assembly. Here are the eight most pervasive mistakes developers make and exactly how to fix them.

The earliest stages of the RAG pipeline are where the most irreversible errors occur.

• 1. Arbitrary Fixed-Character Splitting: Chopping text strictly every 500 characters splits words, table rows, and multi-sentence explanations in half. Fix: Use recursive character or markdown-aware chunking that respects headers and paragraph breaks.
• 2. Relying Solely on Vector Cosine Distance: Embedding models compress meaning into general semantic directions, failing completely on exact part numbers (e.g. `X-800`), error codes, and unique proper nouns. Fix: Implement hybrid search combining BM25 keyword matching with vector embeddings.
• 3. Ignoring Document Metadata: Treating all chunks as an undifferentiated pool makes it impossible to filter by date, department, or user permissions. Fix: Attach structured metadata (author, created date, access level) to every vector row and apply pre-retrieval SQL filtering.
• 4. Omitting a Cross-Encoder Reranker: Bi-encoder vector search is great for fast coarse screening, but pulls false positives. Passing unranked vector chunks directly to the LLM pollutes the prompt. Fix: Deploy a cross-encoder reranker (like Cohere Rerank or BGE) to filter top-30 candidates down to the top 4 pristine passages.

Even if your retrieval engine fetches pristine chunks, poor prompt assembly and lack of conversation state will derail the user experience.

• 5. Context Stuffing (Injecting Too Many Chunks): Pulling 20 chunks into a prompt creates noise and triggers attention diffusion. The model suffers from the "lost-in-the-middle" effect and overlooks the single relevant paragraph. Fix: Quality over quantity — pass only the top 3-5 verified chunks.
• 6. Permissive Prompt Instructions: If you tell an LLM "Use this context to answer the question", but the context doesn’t have the answer, the model feels obligated to produce a response, drawing on pre-trained assumptions to fabricate a plausible hallucination. Fix: Add a strict negative guardrail: "Answer strictly based on the provided context. If the context does not provide the answer, state that the documentation does not contain this information."
• 7. Neglecting Multi-Turn Query Condensation: In chat applications, users ask follow-up questions like "How much does that cost?" If you embed that sentence directly, vector search returns pricing for random products. Fix: Use a lightweight LLM to rewrite the follow-up into a standalone question before running retrieval.
• 8. Lacking Continuous Evaluation Metrics: Tweaking chunk sizes and prompt instructions based on casual "vibe checks" leads to silent regressions. Fix: Maintain a gold-standard evaluation dataset of 100 test questions and measure Context Recall and Answer Faithfulness automatically in CI/CD.

To keep your RAG application healthy over months and years, treat it like an observable software service. Log every user query, the exact chunk IDs retrieved, their similarity scores, the final prompt sent to the LLM, and the user’s reaction (thumbs up/down).

Reviewing the bottom 5% of queries with low user ratings or low retrieval similarity will immediately expose gaps in your documentation, poor chunk boundaries, or vocabulary mismatches.

• Log all intermediate steps: query, rewritten query, retrieved chunk IDs, rerank scores, and final output
• Add user feedback buttons (thumbs up / down) to collect real-world failure cases automatically
• Establish automated regression tests in CI/CD before updating embedding models or prompts
• Audit your document sources regularly to purge deprecated or contradictory files