Latency Budgets for Memory Retrieval
Set workload-specific latency targets, trace the critical path, and compare retrieval changes by answer quality, tail latency and cost.
A memory-retrieval latency budget is the time your application can spend finding context before it misses its response target. Set that target for the interaction you are building. A live voice turn, a support chat and a background research task have different constraints.
Measure the current path before choosing a database or reranker. A fast index does not establish a fast application if query encoding, network requests or model generation dominate the user's wait.
Define what the clock measures
Distinguish time to first useful output from total completion time. For a streaming answer, the first token can arrive before the answer is complete. For a voice agent, end-of-turn detection and speech synthesis add stages that a text-only retrieval benchmark does not measure.
Record at least the request start, retrieval start and end, model start, first output and completion. Trace retries and errors as well as successes. An apparent speed improvement caused by failed requests returning early is not a better user experience.
Trace the stages that actually exist
A vector-based path may encode a query, search an index, rerank candidates and assemble context. A lexical-only path may not generate an embedding. A cache hit may bypass a stage. Instrument the stages present in your application.
Precomputing document embeddings removes repeated document encoding. It does not generally remove query encoding for a new semantic-search query. A reusable query embedding or an exact cache hit is a separate optimization.
Separate ingestion and memory formation from retrieval. Slow processing of a newly uploaded file affects freshness; a slow search over ready data affects request latency. Both matter, but they need different instrumentation and fixes.
Use an explicit example budget
Suppose a product chooses a 500 ms allowance for its retrieval path. An illustrative allocation could be:
| Stage | Planning allowance |
|---|---|
| Network and edge overhead | 50 ms |
| Orchestration | 80 ms |
| Primary retrieval | 120 ms |
| Reranking and result assembly | 100 ms |
| Remaining headroom | 150 ms |
| Total | 500 ms |
This example allocates the full 500 ms to retrieval. Budget answer generation separately, and adjust the stage allowances after measuring your own path.
A timeout or retry policy must fit the overall deadline. Reserving headroom does not guarantee that a retry can finish. Decide whether the user should receive a partial answer, a clear failure or a later result when the deadline expires.
Measure the tail without misusing percentiles
P50 is the median request duration. P95 is a threshold at or below which approximately 95% of observations fall, under the stated measurement method. The remaining slow requests can still matter, especially when a workflow makes many calls.
Report percentiles with the sample size, observation window, concurrency and failure rate. A small sample cannot characterize rare tail events reliably. Do not add each stage's P95 and label the sum the end-to-end P95; slow stages may occur on different requests or overlap. Trace complete requests to measure the actual distribution.
For sequential work, elapsed durations accumulate. For independent parallel work, the critical path depends on the slowest required branch and orchestration overhead. Parallelism can also increase load and worsen tail behavior. Measure the implementation you actually ship.
Treat optimizations as experiments
| Change | Hypothesis | Failure to watch |
|---|---|---|
| Smaller candidate set | Less retrieval and reranking work | Supporting evidence falls outside the set |
| Reranking selected candidates | Better evidence ordering | Added latency without better answers |
| Query embedding cache | Avoid repeated encoding | Wrong reuse after model or input changes |
| Scoped result cache | Reuse repeated evidence | Stale content or cross-user leakage |
| Prefetch | Finish useful retrieval before it is needed | Wasted calls or wrong predicted context |
A smaller first-stage candidate set may save time but leave the reranker without the evidence it needs. Compare changes with a fixed set of questions and an explicit quality criterion.
Cache keys need the relevant scope, permissions, source version and retrieval configuration. Correcting or deleting a memory should invalidate derived responses where required. A five-millisecond cache hit is not useful if it returns another customer's data or an obsolete policy.
Investigate load and maintenance effects
Test cold and warm caches, realistic concurrency, large tenants and ongoing writes. Capture queueing, rate limits, timeouts and retries. A burst at session start can overload a shared dependency even when a single-user test looks fast.
Index maintenance behavior depends on the database and configuration. Use the engine's metrics and documentation to identify whether maintenance is contributing to a regression.
Keep benchmark comparisons matched on dataset, dimensions, filters, recall target, hardware, concurrency and cache state. An isolated vendor number does not establish the fastest database for your application.
Apply the same checks to Supermemory
Use the documented search modes and inspect the returned evidence. Supermemory's “hybrid” mode returns memories and document chunks, so choose it when the question needs both kinds of evidence.
Measure the API round trip from your deployment region. Then measure the complete agent response with and without the added context. Record retrieval and model timings separately to see how each contributes to the user's wait.
The memory evaluation guide and retrieval debugging guide provide complementary checks. Keep quality, failure rate and cost alongside latency so the optimization has a defensible outcome.
Run a Supermemory pilot with representative queries and concurrency. Select the retrieval budget from the measured user experience, then use traces to identify the next change worth testing.
Frequently asked questions
Is there one latency target for every AI agent?
No. Set a target for the interaction and measure its complete critical path, including generation and any voice stages.
Does precomputing document embeddings eliminate query encoding?
No. A new semantic-search query generally still needs an embedding unless a suitable cached embedding or another retrieval path is used.
Does reranking always make retrieval faster?
No. Reranking adds work. It may improve evidence selection, but its quality and latency tradeoffs must be measured.