How to build a production RAG application from scratch

At its architectural core, Retrieval-Augmented Generation (RAG) is straightforward: it bridges the gap between what an LLM memorized during training and the proprietary or dynamic documents you want it to know.

While tutorials often combine the process into a single library call, a production-grade system decouples into four distinct, independent stages: Document Ingestion & Cleaning, Chunking & Semantic Splitting, Vector Embedding & Storage, and the Retrieval & Generation Loop.

Understanding how data flows between these stages is what separates fragile hackathon prototypes from resilient enterprise software.

The ingestion stage converts raw documents — such as PDFs, markdown files, HTML pages, or database records — into clean plain text. PDFs are particularly treacherous: headers, footers, page numbers, and multi-column layouts frequently corrupt sentence flow if parsed with basic text extractors.

Once text is cleaned, it must be chunked. A common beginner mistake is splitting text every 500 characters. If a critical definition starts at character 480 and finishes at character 550, the definition is cleaved in two, destroying its semantic vector representation.

Modern RAG architectures use recursive character splitting or semantic boundary chunking. They split by markdown headers (`#`, `##`), then by double line breaks (paragraphs), and finally by sentence punctuation. This ensures each chunk represents a complete, cohesive thought.

• Target chunk sizes between 250 and 500 tokens for optimal embedding model density
• Incorporate a 10% to 15% sliding window overlap between chunks to prevent boundary information loss
• Attach metadata (document ID, section title, page number, created date) to every chunk for query-time filtering
• Filter out boilerplate: remove repetitive copyright notices and navigation menus prior to embedding

An embedding model takes a text chunk and transforms it into a vector — an array of floating-point numbers (e.g. 1,536 dimensions) representing its location in semantic space. Words and phrases with similar meanings end up geometrically close to one another.

To store and search these vectors, you do not need an esoteric, expensive cloud vector service. PostgreSQL with the `pgvector` extension is one of the most reliable and battle-tested vector stores available.

With pgvector, you store your text chunks, metadata, and embeddings in standard relational tables. You can index vectors using HNSW (Hierarchical Navigable Small World) for sub-millisecond approximate nearest neighbor searches, while combining vector similarity with standard SQL `WHERE` clauses.

When a user submits a question, the application passes the prompt through the identical embedding model used during ingestion to obtain a query vector.

The system searches the vector database for the top-k (usually 3 to 5) most similar chunks. These chunks are formatted into a clean context block and injected into the LLM system prompt alongside strict grounding instructions.

The system prompt explicitly commands the model: "Answer the question using ONLY the provided context. If the answer cannot be determined from the context, state that the information is unavailable."

• Always include citation instructions: require the model to cite chunk indices or source document titles
• Stream LLM output tokens to the client to keep perceived latency low
• Log the retrieved chunks alongside user questions to monitor retrieval quality over time
• Implement defensive prompt guardrails to prevent user prompts from overriding retrieval constraints