How to Evaluate Agent Memory with MemoryBench
Run a reproducible memory evaluation, interpret accuracy, latency, and context tokens, and add product-specific regression cases.
A memory evaluation should show whether the agent answered correctly, what evidence it retrieved, and what that result cost in time and context. MemoryBench provides a common harness for running supported memory providers against benchmark datasets. Your product still needs a separate set of cases for its own users and failure modes.
The most useful first run is small enough to inspect. A large score table is hard to interpret when you have not checked that ingestion, retrieval, and judging are behaving as intended.
Start from the actual repository
Use the official MemoryBench repository, record the commit, and follow its configuration instructions. Avoid installing a similarly named package and assuming it exposes the same API.
git clone https://github.com/supermemoryai/memorybench.git
cd memorybench
bun install
git rev-parse HEAD
bun run src/index.ts helpConfigure the provider and model credentials required for your run using the repository's documented environment settings. Run with synthetic or appropriately approved data; the harness sends benchmark content to the configured services and incurs their normal usage costs.
The repository documents this limited LoCoMo run:
bun run src/index.ts run -p supermemory -b locomo -l 10Ten questions provide a smoke test of the setup. They do not establish a reliable ranking between providers.
What should an agent-memory benchmark measure?
Measure supported answers, retrieval evidence, latency, and context consumption separately. Include the denominator: 43 correct answers out of 50 evaluated questions is 86% accuracy on that set. Also report setup failures and unanswered requests so a clean percentage cannot hide failed completion.
Use public datasets for repeatable comparisons and application-specific cases for release decisions. For example, LongMemEval tests recall and reasoning over conversation history, while a customer-support pilot must also check whether the agent repeats a failed troubleshooting step. Neither test substitutes for the other.
Inspect the pipeline before trusting the total
MemoryBench separates ingestion, indexing, search, answering, and evaluation. Check the output of each stage for a few questions.
If the required source never reached the provider, a poor answer is not evidence of a retrieval problem. If it reached the provider but was still being processed, record the readiness behavior. If the right evidence reached the answering model, investigate prompt construction or reasoning before tuning the index.
Preserve the run configuration with the results. A screenshot of the final score cannot explain a later regression.
Read MemScore as three measurements
The repository describes MemScore as an accuracy, latency, and context-token triple. It does not collapse precision, coherence, temporal quality, and latency into one weighted score.
Keep those dimensions separate when making a decision. A system may improve answer quality by retrieving more context, or reduce latency at the expense of useful evidence. Neither tradeoff is inherently right for every application.
For your own reporting, also show the number of evaluated questions and failures. A result of 43 correct out of 50 evaluated questions is 86% accuracy; it does not mean “86% reliable in production.” The sample and test conditions define what was measured.
Compare providers under the same conditions
Use the same dataset revision, question set, answering model, judge, and scoring rules. Record provider-specific retrieval settings rather than assuming identical defaults.
Separate setup failures from incorrect answers, but do not silently discard them. Decide whether the comparison is measuring quality after successful ingestion or end-to-end completion including infrastructure errors, and state that choice.
When repeating runs, preserve question IDs and configuration. Model variability can move scores, especially on a small sample. Inspect the cases that change instead of treating every small difference as a meaningful improvement.
Add a product-specific regression suite
A support agent needs cases about prior attempted fixes. A coding agent needs cases about decisions that remain valid after the repository changes. A research assistant needs source provenance and a way to retract outdated evidence.
For each case, record the source history, allowed scope, question, expected behavior, and evidence required to justify the answer. Include negative cases where the correct response is to ask for clarification or decline to infer a fact.
Keep isolation and deletion as explicit pass-or-fail checks. Do not average a cross-tenant disclosure into a broad quality score where better performance elsewhere can conceal it.
Use a failure to choose the next experiment
Change one relevant part of the pipeline, rerun the affected cases, and then run the broader regression set. A retrieval change should be evaluated for both the cases it fixes and the cases it disrupts.
Use the LongMemEval guide to understand that dataset's scope and the memory debugging guide to trace failures through the application. The outcome of an evaluation is a reproducible decision about what to improve next.
Turn the evaluation plan into a run: get MemoryBench, follow the repository’s setup instructions, and start with a supported provider and dataset. Save the configuration and outputs as your baseline before changing the memory pipeline.