JobsDart LogoJobsDart
AI EngineeringVerified & Peer Reviewed

How to build a production-ready RAG chatbot for your website

Single-turn RAG is simple, but building a conversational chatbot requires handling multi-turn follow-ups, condensed query rewriting, token streaming, and clear citations.

J
JobsDart Editorial
4 min read

Key Takeaways

  • Naive conversational RAG fails on pronouns ("What did they do next?") because embedding models cannot infer context without query condensation.
  • Query condensation uses a fast model to rewrite conversational follow-ups into standalone, self-contained search queries before retrieval.
  • Chat history must be separated: past conversation turns provide conversational context, while retrieved chunks provide factual grounding.
  • Streaming answers via Server-Sent Events (SSE) cuts perceived time-to-first-token to under 300 milliseconds.
  • Always provide interactive source badges in the UI so users can verify citations without disrupting chat flow.

The query condensation step: resolving pronouns and history

The critical component in a conversational RAG chatbot is the Query Reformulator (often termed query condensation or contextualization).

Before the chatbot executes a search against your vector database, it passes the recent chat history (the last 3-4 turns) and the user’s new message to a fast, lightweight LLM with a strict prompt: "Given the chat history and the latest user message, rephrase the message into a complete, standalone question that can be understood without the prior conversation."

When the user asks "How much does it cost?", the query condenser outputs: "What is the pricing and cost of the Enterprise Plan?" This standalone question is then embedded and retrieved against the knowledge base with 100% relevance.

Query condensation examples in multi-turn chat
Prior ContextUser Follow-upCondensed Search Query Generated
Discussing refund policy for damaged goodsHow many days do I have?What is the return window timeframe for damaged products?
Talking about remote developer positions in GermanyDo you sponsor visas for it?Does the company offer visa sponsorship for remote developer roles in Germany?
Explaining health insurance benefitsWhat about dental?What dental coverage and benefits are included in the company health plan?

Structuring prompt context: conversation history vs retrieved chunks

A common architectural bug in custom chatbots is concatenating the retrieved document chunks directly into the user message history. This confuses the model, causing it to treat reference documentation as past conversational banter.

Keep the two context streams strictly separated in the model prompt:

  • System Message: Contains the agent persona, tone guidelines, and strict grounding instructions: "Answer only using the provided Reference Context. If unsure, admit it."
  • Reference Context Block: Contains the retrieved chunks with unique identifiers (`[Doc 1]`, `[Doc 2]`).
  • Conversation History: Contains the clean, chronological sequence of prior user questions and assistant answers (omitting past chunks to save tokens).
  • Latest User Message: The current question triggering the response.

Streaming responses and interactive source citations

A customer waiting five seconds in front of a frozen text box assumes the chatbot has crashed. Production chatbots use Server-Sent Events (SSE) or WebSockets to stream generated tokens immediately as they leave the inference engine.

At the start of the stream, send a JSON metadata packet containing the retrieved citation titles, URLs, and similarity scores. Then stream the raw markdown text tokens. The front-end renders the text progressively with a markdown parser, replacing citation tags like `[Doc 1]` with interactive tooltip badges that preview the exact quoted text.

  • Stream tokens via standard HTTP Server-Sent Events (SSE) for maximum firewall and proxy compatibility
  • Render citation badges as hoverable tooltips so users can verify facts without leaving the chat interface
  • Implement rate limiting and token consumption guards to prevent abusive multi-thousand token queries
  • Add thumbs-up/thumbs-down feedback widgets on every assistant response to build your evaluation dataset

Frequently Asked Questions

How do you build a RAG chatbot?

To build a RAG chatbot: 1) Index your documentation into a vector store, 2) Use a query condensation model to rewrite conversational follow-ups into standalone queries, 3) Retrieve matching chunks, 4) Inject both conversation history and retrieved chunks into an LLM prompt, and 5) Stream the response with citations.

Why do naive RAG chatbots fail on follow-up questions?

They fail because follow-up questions use pronouns like "it", "they", or "the second option". Embedding these pronoun-heavy questions without conversational context returns irrelevant search chunks.

What is query condensation in conversational AI?

Query condensation is a preprocessing step where an LLM analyzes the chat history and the user’s new question to produce a fully qualified, standalone search prompt that contains all necessary entities and context.

How do you handle citations in a RAG chatbot?

Number your retrieved chunks (e.g. `[Doc 1]`, `[Doc 2]`) in the prompt and instruct the model to reference them in its answer. On the client, parse these markers into clickable badges that display the source title and excerpt.

How do you stream answers from a RAG chatbot?

Use Server-Sent Events (SSE) to send chunks of generated text to the browser in real time, reducing the user’s perceived latency to just a few hundred milliseconds.

Further Reading & Primary Sources

JobsDart First-Party Tools

Put This Into Practice with JobsDart Tools

Test and apply the principles from this guide directly using our dedicated career tools. Designed to eliminate guesswork and surface verified opportunities.

JobsDart Verified Job Discovery

Direct Recruiter Review

Explore verified global, remote, and corporate tech openings with direct hiring team review and real-time application tracking.

Browse Active Roles

JobsDart Free ATS Resume Checker

Instant Analysis

Validate that your resume formatting and keyword taxonomy match enterprise hiring standards before submitting.

Check Resume Match
J

JobsDart Editorial

Platform Research & Technical Editorial Board

Researched and audited by the JobsDart Technical Career Advisory Team. All guides, ATS benchmarks, and career roadmaps are tested against empirical hiring data, enterprise ATS parsers, and verified recruiter workflows.

Reviewed under JobsDart Technical Editorial StandardsLearn more about our methodology →
All Career Guides