[Blog](https://supermemory.ai/blog) · [Learning](https://supermemory.ai/blog/tag/learning)

# How to Build a RAG Chatbot: Retrieval, Citations, and Evaluation

Build a RAG chatbot around authorized retrieval, traceable evidence, and answer-quality tests. Includes a runnable local retrieval baseline.

By Shardul Mane March 29, 2026 · 5 min read 

![How to Build a RAG Chatbot: Retrieval, Citations, and Evaluation](https://supermemory.ai/_astro/cover.DBZIK_k1_v2gbl.webp)

Retrieval-augmented generation (RAG) answers a question using information fetched from an external source. A RAG chatbot needs a searchable corpus, a retrieval step, and a model that receives the retrieved evidence. A useful production implementation also needs permission checks, document versions, citations, and a way to decline questions the corpus cannot answer.

Start with a small set of real questions and known supporting passages. Build a baseline that can retrieve those passages before adding query rewriting, reranking, or graph traversal. Otherwise, every extra stage makes it harder to tell why an answer improved or failed.

## What does a RAG chatbot actually do?

The ingestion path turns source material into searchable records. The request path selects records for a particular user and question, then generates an answer from that evidence.

| Stage                  | Output                                   | Failure to look for                                          |
| ---------------------- | ---------------------------------------- | ------------------------------------------------------------ |
| Parse a document       | Text with source and section identifiers | A table loses its headers or a scanned page produces no text |
| Split and index        | Searchable chunks with metadata          | A condition and its exception land in disconnected chunks    |
| Authorize and retrieve | Permitted candidate passages             | Results include another customer's document                  |
| Select context         | A bounded set of relevant passages       | Repeated passages consume the evidence budget                |
| Generate and cite      | An answer tied to specific sources       | The answer adds a condition the sources never stated         |

Store enough information to trace an answer back to a document version. A link to the homepage of a large knowledge base is not a useful citation for a specific refund policy.

## Build a local retrieval baseline first

This Python example uses SQLite FTS5 to show the retrieval boundary without credentials or a model call. It searches a tiny fictional corpus for an exact term. It is a runnable lexical baseline, not semantic retrieval or a complete chatbot. Use a Python installation whose SQLite includes FTS5.

```
import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE VIRTUAL TABLE docs USING fts5(tenant UNINDEXED, source UNINDEXED, body)")
db.executemany("INSERT INTO docs VALUES (?, ?, ?)", [
    ("acme", "refunds-v2#annual", "Annual refunds require approval within 14 days."),
    ("acme", "shipping-v1", "Shipping normally takes five business days."),
    ("other", "refunds-private", "Refunds are approved automatically."),
])

def retrieve(tenant, term):
    if not term or not term.isalnum():
        raise ValueError("This demo accepts one alphanumeric search term")
    return db.execute(
        "SELECT source, body FROM docs WHERE docs MATCH ? AND tenant = ? "
        "ORDER BY bm25(docs) LIMIT 3",
        ('"' + term + '"', tenant),
    ).fetchall()

assert retrieve("acme", "refunds") == [
    ("refunds-v2#annual", "Annual refunds require approval within 14 days.")
]
assert retrieve("acme", "warranty") == []
print(retrieve("acme", "refunds"))
```

The important behavior is that the authorized scope participates in the query. In an application, derive that scope from the authenticated session; do not accept an arbitrary tenant value from the browser. Parameterization also does not make a search language disappear: this example deliberately restricts its input instead of accepting unrestricted FTS expressions.

Next, replace the tiny corpus with representative document sections. Preserve source IDs and permissions. Add semantic search when paraphrases become an observed failure, using the same questions to compare results. The [hybrid search guide](https://supermemory.ai/blog/hybrid-search-guide/) explains how to combine lexical and semantic candidates.

## Add generation after retrieval is inspectable

Pass selected passages as evidence with explicit source labels. Tell the model to answer the question using that evidence and identify when the evidence is insufficient. Keep the user's question separate from quoted document content; a document may contain instructions that the chatbot should treat as data.

A practical response contract contains an answer, its supporting source IDs, and an evidence status such as sufficient, partial, or missing. Check that every emitted source ID belongs to the retrieved set. That catches invented citation identifiers, although it does not prove that the cited passage supports every claim. Evaluate support at the claim level too.

For “Can I get an annual refund after 20 days?”, the example policy establishes a 14-day approval window. It does not establish every exception or the caller's eligibility. The chatbot should explain the stated window and the missing information, rather than manufacture a complete refund decision.

## Choose improvements by the failure they fix

* **Chunking:** change boundaries when a relevant passage is incomplete. See [text chunking strategies](https://supermemory.ai/blog/text-chunking-strategies-for-rag/).
* **Hybrid retrieval:** add an exact-term path when product codes or names are missed, or a semantic path when wording differs.
* **Reranking:** try it when useful evidence enters the candidate set but falls below the context cutoff.
* **Query rewriting:** resolve ambiguous follow-ups using the conversation, while preserving identifiers and permissions.
* **Graph retrieval:** consider it when the question requires explicit relationship paths. It is not a prerequisite for every knowledge base.

Anthropic's [contextual retrieval work](https://www.anthropic.com/engineering/contextual-retrieval) is an example of enriching chunks with document context and evaluating retrieval changes. Compare enrichment against your existing retrieval path using the same labeled questions.

## Evaluate retrieval and answers separately

Create a small release fixture that includes exact terms, paraphrases, questions needing two passages, outdated documents, inaccessible documents, and unanswerable questions. For each question, record acceptable evidence and expected answer behavior.

Measure whether supporting evidence appears within the retrieved set, whether the final answer uses it correctly, and whether citations resolve to the right version. Also record latency, input tokens, and failure rate. A correct answer produced from the model's prior knowledge can hide broken retrieval, so inspect the evidence path directly.

Change one major retrieval component at a time. Compare against a fixed baseline and keep the difficult cases in the regression suite. Report answer accuracy for your own corpus and question set.

## Where does persistent memory fit?

RAG can retrieve documents stored across sessions. Whether the app also remembers user preferences, previous decisions, and corrections depends on what it stores and retrieves. Persistent storage alone does not decide which facts remain current.

Use a [persistent knowledge-base workflow](https://supermemory.ai/blog/building-your-own-ai-knowledge-base-with-supermemory/) for uploaded documents, and a [separate memory lifecycle](https://supermemory.ai/blog/memory-lifecycle-retention-corrections-deletion/) for user-specific context. Keep current account permissions and transactional state in their authoritative systems.

The first milestone is modest and concrete: an authorized user can ask a known question in a fresh session, receive a supported answer with the correct citation, and get an honest “not enough evidence” when the source is missing.

To try a managed retrieval path, [start with Supermemory](https://console.supermemory.ai/) and a small set of documents you can verify by hand. Run the same questions against your baseline, then compare the retrieved evidence and citations before expanding the corpus.

## Other posts.

1. [We're open sourcing the company brain. Here's how we designed the multiplayer harness Company Brain is now open source. A walkthrough of the multiplayer harness behind its Slack experience, from proactivity and memory boundaries to approvals and recovery. NewsSep 25, 2026 ](https://supermemory.ai/blog/open-sourcing-company-brain)
2. [Jev changes a lot in memory & context engineering. Here's exactly how. We tested Jev across reranking, chunking, observation, and harness decisions. Here is where fast decision models help memory systems, where they cost more, and where they still fall short. EngineeringSep 24, 2026 ](https://supermemory.ai/blog/jev-memory-context-engineering)
3. [I reverse-engineered Instinct's memory. Here's exactly how it works Instinct keeps its memory as git-tracked markdown files, found with grep rather than vectors. Here is the whole system as far as black-box probing can reconstruct it, and how to rebuild it on supermemory in about 60 lines. EngineeringSep 20, 2026 ](https://supermemory.ai/blog/reverse-engineering-instinct-memory)
4. [An update to supermemory We've discontinued the supermemory company brain and Nova. Everyone who was charged has been refunded, our MCP and plugins continue to run, and we're going all in on the memory engine. NewsSep 10, 2026 ](https://supermemory.ai/blog/an-update-to-supermemory)
5. [Scaling Conversations: How Adapta Grew Usage Without Losing Context Adapta added Supermemory as a persistent memory layer so every conversation keeps its context — letting the team scale usage without losing the thread. Case StudyJun 12, 2026 ](https://supermemory.ai/blog/adapta-scaling-conversations)
6. [How Chatarmin Ditched RAG and Went Memory-Only with Supermemory Chatarmin replaced a heavy RAG pipeline with Supermemory's memory layer — cutting average AI response time from 40s to 12s and token usage by 40–50%. Case StudyJun 10, 2026 ](https://supermemory.ai/blog/chatarmin-memory-only)
7. [SMFS: making agentic retrieval 55% cheaper AND more accurate We launched SMFS.ai (Supermemory Filesystem) a few weeks ago, with a simple bet: We can redesign the filesystem specifically for agents, with special files, structures, and commands that it can use for it's tasks. Today, SMFS is used by hundreds of companies to power their agents. EngineeringMay 28, 2026 ](https://supermemory.ai/blog/smfs-making-agentic-retrieval-55-cheaper-and-more-accurate)
8. [Introducing Dynamic Dreaming: supermemory now connects the dots, for you. Dreaming is magical. TLDR: We're launching Dynamic Dreaming in supermemory today, which automatically works if you're using supermemory in any way - API, OpenClaw, Hermes agent, etc. EngineeringMay 25, 2026 ](https://supermemory.ai/blog/introducing-dynamic-dreaming-supermemory-now-connects-the-dots-for-you)
9. [Dear reader, we just made supermemory insanely cheap... the Context Cloud When I first started building supermemory, I had one goal: To build the best memory system for AI. I would talk to customers, and find out that memory was not the only thing they needed - They were all setting up 7-8 different vendors at the same time. EngineeringMay 18, 2026 ](https://supermemory.ai/blog/dear-reader-we-just-made-supermemory-insanely-cheap-the-context-cloud)
10. [Introducing @supermemory/tools v2.0.0 Today we're releasing v2.0.0\. This release unifies the API across all agents sdk integrations from AI SDK to Mastra, makes conversation identity a first-class concept, and ships with memory saving on by default. EngineeringApr 27, 2026 ](https://supermemory.ai/blog/introducing-supermemory-tools-v2-0-0)
11. [Solving the Precision-Recall Tradeoff: Search Result Aggregation When you're building memory for AI, search is your foundational layer. The way search generally works is straightforward: the user defines a query, and then sets a limit (top-K) on how many search results they want returned. Usually, this is set to 10 or 20. EngineeringApr 5, 2026 ](https://supermemory.ai/blog/solving-the-precision-recall-tradeoff-search-result-aggregation)
12. [OpenClaw Memory Problems: Why It Forgets and How to Fix It (2026) TLDR: Today, we are releasing a new version of our openclaw plugin - https://github.com/supermemoryai/openclaw-supermemory. This post is going to be a bit technical, so bear with me (or bookmark for later!) In this post, I will talk about what we do about OpenClaw memory, and how we fix it. EngineeringFeb 19, 2026 ](https://supermemory.ai/blog/why-everyone-is-complaining-about-openclaws-memory-it-sucks-and-why-supermemory-fixes-it)
13. [Stateful Coding Agents with Memory: Build Long-Running Agents (2026) We built a plugin for Claude Code and OpenCode that gives your coding agent persistent memory. It remembers your preferences, learns your codebase, and never loses context mid-conversation. The result is an agent you can run for months without starting over. EngineeringFeb 18, 2026 ](https://supermemory.ai/blog/infinitely-running-stateful-coding-agents)
14. [Clawd / Molt bot's memory SUCKS. We gave it supermemory. I'm the founder of supermemory. Clawd/Molt bot is blowing up right now, with many, many use cases. I set it up, too, and have been using it through telegram. TLDR: just go to https://supermemory.ai/docs/integrations/clawdbot to set up supermemory for your clawd bot. EngineeringJan 28, 2026 ](https://supermemory.ai/blog/clawd-molt-bots-memory-sucks-we-gave-it-supermemory)
15. [Catch up with our UNFORGETTABLE Launch Week Over the last year, one belief has guided almost everything we’ve built at Supermemory AI becomes meaningfully useful only when it remembers. Memory shouldn’t be something developers rebuild from scratch. It shouldn’t be fragile, expensive, or trapped inside a single tool. NewsJan 4, 2026 ](https://supermemory.ai/blog/catch-up-with-our-unforgettable-launch-week)
16. [Empowering the Next Generation of Founders: Supermemory Startup Program If there’s one thing we’ve learned while building Supermemory, it’s that most startups don’t fail because they didn't build features; they fail when infrastructure slows them down, or they built too slow. NewsDec 31, 2025 ](https://supermemory.ai/blog/empowering-the-next-generation-of-founders-supermemory-startup-program)
17. [Building code-chunk: AST Aware Code Chunking At Supermemory, we're building context engineering infrastructure for AI. A huge part of that is dealing with code: ingesting repos, understanding structure, and making it searchable. The problem is that most code chunking solutions are terrible. We built code-chunk to fix this. EngineeringDec 29, 2025 ](https://supermemory.ai/blog/building-code-chunk-ast-aware-code-chunking)
18. [Supermemory raises $3 million with the best memory engine for LLMs Today, I am excited to announce our first funding round to accelerate our mission of building an interoperable, scalable and reliable memory for LLMs and agents. Memory is one of the hardest challenges in AI right now. NewsOct 6, 2025 ](https://supermemory.ai/blog/supermemory-raises-3-million-and-building-the-best-memory-engine-for-llms)
19. [Mem0 vs Supermemory: Why Scira Switched Scira AI moved its production memory layer from Mem0 to Supermemory. This is what failed, what improved, and how the team evaluated the two systems. Case StudyOct 2, 2025 ](https://supermemory.ai/blog/why-scira-ai-switched)
20. [Never Record Again: How Montra Uses Supermemory to Rethink Video Creation Campbell Baron, the founder of Montra, has been making videos since he was twelve. By thirteen, he was already doing brand work. Today, he’s betting on a very different future for creators: a world where recording is the exception, and most videos are generated from scratch. Case StudyAug 21, 2025 ](https://supermemory.ai/blog/never-record-again-how-montra-uses-supermemory-to-rethink-video-creation)
21. [Unified Memory That Works Where You Work: Your Second Brain With Supermemory Hi everyone, I’m Dhravya, the founder of Supermemory. I want to start with a little story behind why this product means so much to me. You can also skip straight to what it is and how it works below. EngineeringJul 25, 2025 ](https://supermemory.ai/blog/unified-memory-that-works-where-you-work-your-second-brain-with-supermemory)
22. [Supermemory just got faster on PlanetScale What is Supermemory? Supermemory completes the missing part of the LLM puzzle: memory. Just as memory is crucial for human intelligence, it's essential for truly intelligent AI systems. EngineeringJul 18, 2025 ](https://supermemory.ai/blog/supermemory-just-got-faster-on-planetscale)
23. [Faster, smarter, reliable infinite chat: Supermemory IS context engineering. People are obsessed with prompts and prompt engineering. Sure, what you say is important, but what the model knows when you say it is the difference between a stateless text generator and an intelligent AI system. In short, context is the most crucial component. NewsJul 9, 2025 ](https://supermemory.ai/blog/faster-smarter-reliable-infinite-chat-supermemory-is-context-engineering)
24. [We solved AI API interoperability One API to rule them all, One spec to find them, One library to bring them all and in the TypeScript, bind them. When we were building the Infinite Chat API, initially, we only supported the OpenAI format. This was fine, until a lot of our customers started asking for more. EngineeringJul 7, 2025 ](https://supermemory.ai/blog/we-solved-ai-api-interoperability)
25. [The Wow Factor of Memory - How Flow Used Supermemory To Build Smarter, Stickier Products Overview: Flow is a note-taking app built around a bold vision: to create a more personal, context-aware writing experience powered by AI. At the heart of this mission is memory. Case StudyJun 14, 2025 ](https://supermemory.ai/blog/the-wow-factor-of-memory-how-flow-used-supermemory-to-build-smarter-stickier-products)
26. [The UX and technicalities of awesome MCPs Last month, we launched the Supermemory MCP, mostly to test our own infrastructure and get some initial traction. It blew up. To my absolute surprise, the initial launch itself got half a million impressions (!!!). Then, we launched and got #2 on ProductHunt too. EngineeringJun 8, 2025 ](https://supermemory.ai/blog/the-ux-and-technicalities-of-awesome-mcps)
27. [Architecting a memory engine inspired by the human brain Language is at the heart of intelligence, but what truly powers meaningful interaction is memory — the ability to accumulate, recall, and contextualize information over time. Large Language Models (LLMs) have mastered language, but memory remains their Achilles’ heel. EngineeringJun 5, 2025 ](https://supermemory.ai/blog/memory-engine)

## Start building with supermemory.

Memory and continual learning for any model, any harness. Available through our API, plugins, and MCP.

[Build with supermemory ](https://console.supermemory.ai/)