[Blog](https://supermemory.ai/blog) · [Learning](https://supermemory.ai/blog/tag/learning)

# Latency Budgets for Memory Retrieval

Set workload-specific latency targets, trace the critical path, and compare retrieval changes by answer quality, tail latency and cost.

By Shardul Mane May 13, 2026 · 5 min read 

![Budget for memory retrieval latency](https://supermemory.ai/_astro/cover.jU5NWdvn_Z11id53.webp)

A memory-retrieval latency budget is the time your application can spend finding context before it misses its response target. Set that target for the interaction you are building. A live voice turn, a support chat and a background research task have different constraints.

Measure the current path before choosing a database or reranker. A fast index does not establish a fast application if query encoding, network requests or model generation dominate the user's wait.

## Define what the clock measures

Distinguish time to first useful output from total completion time. For a streaming answer, the first token can arrive before the answer is complete. For a voice agent, end-of-turn detection and speech synthesis add stages that a text-only retrieval benchmark does not measure.

Record at least the request start, retrieval start and end, model start, first output and completion. Trace retries and errors as well as successes. An apparent speed improvement caused by failed requests returning early is not a better user experience.

## Trace the stages that actually exist

A vector-based path may encode a query, search an index, rerank candidates and assemble context. A lexical-only path may not generate an embedding. A cache hit may bypass a stage. Instrument the stages present in your application.

Precomputing document embeddings removes repeated document encoding. It does not generally remove query encoding for a new semantic-search query. A reusable query embedding or an exact cache hit is a separate optimization.

Separate ingestion and memory formation from retrieval. Slow processing of a newly uploaded file affects freshness; a slow search over ready data affects request latency. Both matter, but they need different instrumentation and fixes.

## Use an explicit example budget

Suppose a product chooses a 500 ms allowance for its retrieval path. An illustrative allocation could be:

| Stage                         | Planning allowance |
| ----------------------------- | ------------------ |
| Network and edge overhead     | 50 ms              |
| Orchestration                 | 80 ms              |
| Primary retrieval             | 120 ms             |
| Reranking and result assembly | 100 ms             |
| Remaining headroom            | 150 ms             |
| Total                         | 500 ms             |

This example allocates the full 500 ms to retrieval. Budget answer generation separately, and adjust the stage allowances after measuring your own path.

A timeout or retry policy must fit the overall deadline. Reserving headroom does not guarantee that a retry can finish. Decide whether the user should receive a partial answer, a clear failure or a later result when the deadline expires.

## Measure the tail without misusing percentiles

P50 is the median request duration. P95 is a threshold at or below which approximately 95% of observations fall, under the stated measurement method. The remaining slow requests can still matter, especially when a workflow makes many calls.

Report percentiles with the sample size, observation window, concurrency and failure rate. A small sample cannot characterize rare tail events reliably. Do not add each stage's P95 and label the sum the end-to-end P95; slow stages may occur on different requests or overlap. Trace complete requests to measure the actual distribution.

For sequential work, elapsed durations accumulate. For independent parallel work, the critical path depends on the slowest required branch and orchestration overhead. Parallelism can also increase load and worsen tail behavior. Measure the implementation you actually ship.

## Treat optimizations as experiments

| Change                        | Hypothesis                                  | Failure to watch                          |
| ----------------------------- | ------------------------------------------- | ----------------------------------------- |
| Smaller candidate set         | Less retrieval and reranking work           | Supporting evidence falls outside the set |
| Reranking selected candidates | Better evidence ordering                    | Added latency without better answers      |
| Query embedding cache         | Avoid repeated encoding                     | Wrong reuse after model or input changes  |
| Scoped result cache           | Reuse repeated evidence                     | Stale content or cross-user leakage       |
| Prefetch                      | Finish useful retrieval before it is needed | Wasted calls or wrong predicted context   |

A smaller first-stage candidate set may save time but leave the reranker without the evidence it needs. Compare changes with a fixed set of questions and an explicit quality criterion.

Cache keys need the relevant scope, permissions, source version and retrieval configuration. Correcting or deleting a memory should invalidate derived responses where required. A five-millisecond cache hit is not useful if it returns another customer's data or an obsolete policy.

## Investigate load and maintenance effects

Test cold and warm caches, realistic concurrency, large tenants and ongoing writes. Capture queueing, rate limits, timeouts and retries. A burst at session start can overload a shared dependency even when a single-user test looks fast.

Index maintenance behavior depends on the database and configuration. Use the engine's metrics and documentation to identify whether maintenance is contributing to a regression.

Keep benchmark comparisons matched on dataset, dimensions, filters, recall target, hardware, concurrency and cache state. An isolated vendor number does not establish the fastest database for your application.

## Apply the same checks to Supermemory

Use the documented [search modes](https://supermemory.ai/docs/recall/search) and inspect the returned evidence. Supermemory's “hybrid” mode returns memories and document chunks, so choose it when the question needs both kinds of evidence.

Measure the API round trip from your deployment region. Then measure the complete agent response with and without the added context. Record retrieval and model timings separately to see how each contributes to the user's wait.

The [memory evaluation guide](https://supermemory.ai/blog/build-agent-memory-eval-harness/) and [retrieval debugging guide](https://supermemory.ai/blog/debugging-agent-memory-retrieval-postmortem/) provide complementary checks. Keep quality, failure rate and cost alongside latency so the optimization has a defensible outcome.

[Run a Supermemory pilot](https://console.supermemory.ai/) with representative queries and concurrency. Select the retrieval budget from the measured user experience, then use traces to identify the next change worth testing.

## Frequently asked questions

### Is there one latency target for every AI agent?

No. Set a target for the interaction and measure its complete critical path, including generation and any voice stages.

### Does precomputing document embeddings eliminate query encoding?

No. A new semantic-search query generally still needs an embedding unless a suitable cached embedding or another retrieval path is used.

### Does reranking always make retrieval faster?

No. Reranking adds work. It may improve evidence selection, but its quality and latency tradeoffs must be measured.

## Other posts.

1. [We're open sourcing the company brain. Here's how we designed the multiplayer harness Company Brain is now open source. A walkthrough of the multiplayer harness behind its Slack experience, from proactivity and memory boundaries to approvals and recovery. NewsSep 25, 2026 ](https://supermemory.ai/blog/open-sourcing-company-brain)
2. [Jev changes a lot in memory & context engineering. Here's exactly how. We tested Jev across reranking, chunking, observation, and harness decisions. Here is where fast decision models help memory systems, where they cost more, and where they still fall short. EngineeringSep 24, 2026 ](https://supermemory.ai/blog/jev-memory-context-engineering)
3. [I reverse-engineered Instinct's memory. Here's exactly how it works Instinct keeps its memory as git-tracked markdown files, found with grep rather than vectors. Here is the whole system as far as black-box probing can reconstruct it, and how to rebuild it on supermemory in about 60 lines. EngineeringSep 20, 2026 ](https://supermemory.ai/blog/reverse-engineering-instinct-memory)
4. [An update to supermemory We've discontinued the supermemory company brain and Nova. Everyone who was charged has been refunded, our MCP and plugins continue to run, and we're going all in on the memory engine. NewsSep 10, 2026 ](https://supermemory.ai/blog/an-update-to-supermemory)
5. [Scaling Conversations: How Adapta Grew Usage Without Losing Context Adapta added Supermemory as a persistent memory layer so every conversation keeps its context — letting the team scale usage without losing the thread. Case StudyJun 12, 2026 ](https://supermemory.ai/blog/adapta-scaling-conversations)
6. [How Chatarmin Ditched RAG and Went Memory-Only with Supermemory Chatarmin replaced a heavy RAG pipeline with Supermemory's memory layer — cutting average AI response time from 40s to 12s and token usage by 40–50%. Case StudyJun 10, 2026 ](https://supermemory.ai/blog/chatarmin-memory-only)
7. [SMFS: making agentic retrieval 55% cheaper AND more accurate We launched SMFS.ai (Supermemory Filesystem) a few weeks ago, with a simple bet: We can redesign the filesystem specifically for agents, with special files, structures, and commands that it can use for it's tasks. Today, SMFS is used by hundreds of companies to power their agents. EngineeringMay 28, 2026 ](https://supermemory.ai/blog/smfs-making-agentic-retrieval-55-cheaper-and-more-accurate)
8. [Introducing Dynamic Dreaming: supermemory now connects the dots, for you. Dreaming is magical. TLDR: We're launching Dynamic Dreaming in supermemory today, which automatically works if you're using supermemory in any way - API, OpenClaw, Hermes agent, etc. EngineeringMay 25, 2026 ](https://supermemory.ai/blog/introducing-dynamic-dreaming-supermemory-now-connects-the-dots-for-you)
9. [Dear reader, we just made supermemory insanely cheap... the Context Cloud When I first started building supermemory, I had one goal: To build the best memory system for AI. I would talk to customers, and find out that memory was not the only thing they needed - They were all setting up 7-8 different vendors at the same time. EngineeringMay 18, 2026 ](https://supermemory.ai/blog/dear-reader-we-just-made-supermemory-insanely-cheap-the-context-cloud)
10. [Introducing @supermemory/tools v2.0.0 Today we're releasing v2.0.0\. This release unifies the API across all agents sdk integrations from AI SDK to Mastra, makes conversation identity a first-class concept, and ships with memory saving on by default. EngineeringApr 27, 2026 ](https://supermemory.ai/blog/introducing-supermemory-tools-v2-0-0)
11. [Solving the Precision-Recall Tradeoff: Search Result Aggregation When you're building memory for AI, search is your foundational layer. The way search generally works is straightforward: the user defines a query, and then sets a limit (top-K) on how many search results they want returned. Usually, this is set to 10 or 20. EngineeringApr 5, 2026 ](https://supermemory.ai/blog/solving-the-precision-recall-tradeoff-search-result-aggregation)
12. [OpenClaw Memory Problems: Why It Forgets and How to Fix It (2026) TLDR: Today, we are releasing a new version of our openclaw plugin - https://github.com/supermemoryai/openclaw-supermemory. This post is going to be a bit technical, so bear with me (or bookmark for later!) In this post, I will talk about what we do about OpenClaw memory, and how we fix it. EngineeringFeb 19, 2026 ](https://supermemory.ai/blog/why-everyone-is-complaining-about-openclaws-memory-it-sucks-and-why-supermemory-fixes-it)
13. [Stateful Coding Agents with Memory: Build Long-Running Agents (2026) We built a plugin for Claude Code and OpenCode that gives your coding agent persistent memory. It remembers your preferences, learns your codebase, and never loses context mid-conversation. The result is an agent you can run for months without starting over. EngineeringFeb 18, 2026 ](https://supermemory.ai/blog/infinitely-running-stateful-coding-agents)
14. [Clawd / Molt bot's memory SUCKS. We gave it supermemory. I'm the founder of supermemory. Clawd/Molt bot is blowing up right now, with many, many use cases. I set it up, too, and have been using it through telegram. TLDR: just go to https://supermemory.ai/docs/integrations/clawdbot to set up supermemory for your clawd bot. EngineeringJan 28, 2026 ](https://supermemory.ai/blog/clawd-molt-bots-memory-sucks-we-gave-it-supermemory)
15. [Catch up with our UNFORGETTABLE Launch Week Over the last year, one belief has guided almost everything we’ve built at Supermemory AI becomes meaningfully useful only when it remembers. Memory shouldn’t be something developers rebuild from scratch. It shouldn’t be fragile, expensive, or trapped inside a single tool. NewsJan 4, 2026 ](https://supermemory.ai/blog/catch-up-with-our-unforgettable-launch-week)
16. [Empowering the Next Generation of Founders: Supermemory Startup Program If there’s one thing we’ve learned while building Supermemory, it’s that most startups don’t fail because they didn't build features; they fail when infrastructure slows them down, or they built too slow. NewsDec 31, 2025 ](https://supermemory.ai/blog/empowering-the-next-generation-of-founders-supermemory-startup-program)
17. [Building code-chunk: AST Aware Code Chunking At Supermemory, we're building context engineering infrastructure for AI. A huge part of that is dealing with code: ingesting repos, understanding structure, and making it searchable. The problem is that most code chunking solutions are terrible. We built code-chunk to fix this. EngineeringDec 29, 2025 ](https://supermemory.ai/blog/building-code-chunk-ast-aware-code-chunking)
18. [Supermemory raises $3 million with the best memory engine for LLMs Today, I am excited to announce our first funding round to accelerate our mission of building an interoperable, scalable and reliable memory for LLMs and agents. Memory is one of the hardest challenges in AI right now. NewsOct 6, 2025 ](https://supermemory.ai/blog/supermemory-raises-3-million-and-building-the-best-memory-engine-for-llms)
19. [Mem0 vs Supermemory: Why Scira Switched Scira AI moved its production memory layer from Mem0 to Supermemory. This is what failed, what improved, and how the team evaluated the two systems. Case StudyOct 2, 2025 ](https://supermemory.ai/blog/why-scira-ai-switched)
20. [Never Record Again: How Montra Uses Supermemory to Rethink Video Creation Campbell Baron, the founder of Montra, has been making videos since he was twelve. By thirteen, he was already doing brand work. Today, he’s betting on a very different future for creators: a world where recording is the exception, and most videos are generated from scratch. Case StudyAug 21, 2025 ](https://supermemory.ai/blog/never-record-again-how-montra-uses-supermemory-to-rethink-video-creation)
21. [Unified Memory That Works Where You Work: Your Second Brain With Supermemory Hi everyone, I’m Dhravya, the founder of Supermemory. I want to start with a little story behind why this product means so much to me. You can also skip straight to what it is and how it works below. EngineeringJul 25, 2025 ](https://supermemory.ai/blog/unified-memory-that-works-where-you-work-your-second-brain-with-supermemory)
22. [Supermemory just got faster on PlanetScale What is Supermemory? Supermemory completes the missing part of the LLM puzzle: memory. Just as memory is crucial for human intelligence, it's essential for truly intelligent AI systems. EngineeringJul 18, 2025 ](https://supermemory.ai/blog/supermemory-just-got-faster-on-planetscale)
23. [Faster, smarter, reliable infinite chat: Supermemory IS context engineering. People are obsessed with prompts and prompt engineering. Sure, what you say is important, but what the model knows when you say it is the difference between a stateless text generator and an intelligent AI system. In short, context is the most crucial component. NewsJul 9, 2025 ](https://supermemory.ai/blog/faster-smarter-reliable-infinite-chat-supermemory-is-context-engineering)
24. [We solved AI API interoperability One API to rule them all, One spec to find them, One library to bring them all and in the TypeScript, bind them. When we were building the Infinite Chat API, initially, we only supported the OpenAI format. This was fine, until a lot of our customers started asking for more. EngineeringJul 7, 2025 ](https://supermemory.ai/blog/we-solved-ai-api-interoperability)
25. [The Wow Factor of Memory - How Flow Used Supermemory To Build Smarter, Stickier Products Overview: Flow is a note-taking app built around a bold vision: to create a more personal, context-aware writing experience powered by AI. At the heart of this mission is memory. Case StudyJun 14, 2025 ](https://supermemory.ai/blog/the-wow-factor-of-memory-how-flow-used-supermemory-to-build-smarter-stickier-products)
26. [The UX and technicalities of awesome MCPs Last month, we launched the Supermemory MCP, mostly to test our own infrastructure and get some initial traction. It blew up. To my absolute surprise, the initial launch itself got half a million impressions (!!!). Then, we launched and got #2 on ProductHunt too. EngineeringJun 8, 2025 ](https://supermemory.ai/blog/the-ux-and-technicalities-of-awesome-mcps)
27. [Architecting a memory engine inspired by the human brain Language is at the heart of intelligence, but what truly powers meaningful interaction is memory — the ability to accumulate, recall, and contextualize information over time. Large Language Models (LLMs) have mastered language, but memory remains their Achilles’ heel. EngineeringJun 5, 2025 ](https://supermemory.ai/blog/memory-engine)

## Start building with supermemory.

Memory and continual learning for any model, any harness. Available through our API, plugins, and MCP.

[Build with supermemory ](https://console.supermemory.ai/)