How to Build a RAG Chatbot: Retrieval, Citations, and Evaluation

Retrieval-augmented generation (RAG) answers a question using information fetched from an external source. A RAG chatbot needs a searchable corpus, a retrieval step, and a model that receives the retrieved evidence. A useful production implementation also needs permission checks, document versions, citations, and a way to decline questions the corpus cannot answer.

Start with a small set of real questions and known supporting passages. Build a baseline that can retrieve those passages before adding query rewriting, reranking, or graph traversal. Otherwise, every extra stage makes it harder to tell why an answer improved or failed.

What does a RAG chatbot actually do?

The ingestion path turns source material into searchable records. The request path selects records for a particular user and question, then generates an answer from that evidence.

Stage	Output	Failure to look for
Parse a document	Text with source and section identifiers	A table loses its headers or a scanned page produces no text
Split and index	Searchable chunks with metadata	A condition and its exception land in disconnected chunks
Authorize and retrieve	Permitted candidate passages	Results include another customer's document
Select context	A bounded set of relevant passages	Repeated passages consume the evidence budget
Generate and cite	An answer tied to specific sources	The answer adds a condition the sources never stated

Store enough information to trace an answer back to a document version. A link to the homepage of a large knowledge base is not a useful citation for a specific refund policy.

Build a local retrieval baseline first

This Python example uses SQLite FTS5 to show the retrieval boundary without credentials or a model call. It searches a tiny fictional corpus for an exact term. It is a runnable lexical baseline, not semantic retrieval or a complete chatbot. Use a Python installation whose SQLite includes FTS5.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE VIRTUAL TABLE docs USING fts5(tenant UNINDEXED, source UNINDEXED, body)")
db.executemany("INSERT INTO docs VALUES (?, ?, ?)", [
    ("acme", "refunds-v2#annual", "Annual refunds require approval within 14 days."),
    ("acme", "shipping-v1", "Shipping normally takes five business days."),
    ("other", "refunds-private", "Refunds are approved automatically."),
])

def retrieve(tenant, term):
    if not term or not term.isalnum():
        raise ValueError("This demo accepts one alphanumeric search term")
    return db.execute(
        "SELECT source, body FROM docs WHERE docs MATCH ? AND tenant = ? "
        "ORDER BY bm25(docs) LIMIT 3",
        ('"' + term + '"', tenant),
    ).fetchall()

assert retrieve("acme", "refunds") == [
    ("refunds-v2#annual", "Annual refunds require approval within 14 days.")
]
assert retrieve("acme", "warranty") == []
print(retrieve("acme", "refunds"))

The important behavior is that the authorized scope participates in the query. In an application, derive that scope from the authenticated session; do not accept an arbitrary tenant value from the browser. Parameterization also does not make a search language disappear: this example deliberately restricts its input instead of accepting unrestricted FTS expressions.

Next, replace the tiny corpus with representative document sections. Preserve source IDs and permissions. Add semantic search when paraphrases become an observed failure, using the same questions to compare results. The hybrid search guide explains how to combine lexical and semantic candidates.

Add generation after retrieval is inspectable

Pass selected passages as evidence with explicit source labels. Tell the model to answer the question using that evidence and identify when the evidence is insufficient. Keep the user's question separate from quoted document content; a document may contain instructions that the chatbot should treat as data.

A practical response contract contains an answer, its supporting source IDs, and an evidence status such as sufficient, partial, or missing. Check that every emitted source ID belongs to the retrieved set. That catches invented citation identifiers, although it does not prove that the cited passage supports every claim. Evaluate support at the claim level too.

For “Can I get an annual refund after 20 days?”, the example policy establishes a 14-day approval window. It does not establish every exception or the caller's eligibility. The chatbot should explain the stated window and the missing information, rather than manufacture a complete refund decision.

Choose improvements by the failure they fix

• Chunking: change boundaries when a relevant passage is incomplete. See text chunking strategies.
• Hybrid retrieval: add an exact-term path when product codes or names are missed, or a semantic path when wording differs.
• Reranking: try it when useful evidence enters the candidate set but falls below the context cutoff.
• Query rewriting: resolve ambiguous follow-ups using the conversation, while preserving identifiers and permissions.
• Graph retrieval: consider it when the question requires explicit relationship paths. It is not a prerequisite for every knowledge base.

Anthropic's contextual retrieval work is an example of enriching chunks with document context and evaluating retrieval changes. Compare enrichment against your existing retrieval path using the same labeled questions.

Evaluate retrieval and answers separately

Create a small release fixture that includes exact terms, paraphrases, questions needing two passages, outdated documents, inaccessible documents, and unanswerable questions. For each question, record acceptable evidence and expected answer behavior.

Measure whether supporting evidence appears within the retrieved set, whether the final answer uses it correctly, and whether citations resolve to the right version. Also record latency, input tokens, and failure rate. A correct answer produced from the model's prior knowledge can hide broken retrieval, so inspect the evidence path directly.

Change one major retrieval component at a time. Compare against a fixed baseline and keep the difficult cases in the regression suite. Report answer accuracy for your own corpus and question set.

Where does persistent memory fit?

RAG can retrieve documents stored across sessions. Whether the app also remembers user preferences, previous decisions, and corrections depends on what it stores and retrieves. Persistent storage alone does not decide which facts remain current.

Use a persistent knowledge-base workflow for uploaded documents, and a separate memory lifecycle for user-specific context. Keep current account permissions and transactional state in their authoritative systems.

The first milestone is modest and concrete: an authorized user can ask a known question in a fresh session, receive a supported answer with the correct citation, and get an honest “not enough evidence” when the source is missing.

To try a managed retrieval path, start with Supermemory and a small set of documents you can verify by hand. Run the same questions against your baseline, then compare the retrieved evidence and citations before expanding the corpus.