What Makes Agent Memory Useful? Test the User Outcome

Useful agent memory reduces repeated explanation while keeping answers current, relevant, and under the user's control. Saving more conversation text is not itself an improvement. Evaluate whether the assistant completes a recurring task better and whether users can correct the information that drives its behavior.

Choose an outcome a user can recognize

For a writing assistant, the outcome might be applying a confirmed style preference in a later session. For a support assistant, it might be remembering a prior troubleshooting step without repeating it. For a research assistant, it might be continuing from a verified decision with the source still available.

Write the task before designing the memory. This prevents a system from accumulating facts that are easy to extract but irrelevant to the product. A user's favorite color is not useful context for every workflow simply because it is easy to store.

Test more than successful recall

Use a compact acceptance set: the information is remembered in a new session; a correction replaces the old behavior; a temporary exception stays temporary; another user's information never appears; and removed information no longer influences the answer.

Add a case where no relevant memory exists. The assistant should ask for context or answer within its evidence instead of inventing a remembered relationship. Include a case where the memory is relevant but the current user instruction should take precedence.

Observe burden as well as accuracy

Track repeated explanations, unnecessary memory references, correction effort, and task completion. A technically accurate recollection can still feel intrusive if the assistant brings up unrelated personal information.

Show users enough context to understand why a preference was applied. A concise “based on your saved preference” with a way to inspect or change it can be more useful than an opaque personalized response. Match the interface to the sensitivity and importance of the remembered information.

Expand only when the benefit survives exceptions

Review failures by capture, retrieval, context assembly, and answer behavior. Fix the stage responsible rather than adding more facts to the store. Maintain the acceptance set as the product changes.

For the product interface, the decision-history UI guide shows how to make remembered information inspectable.

The personalization guide covers collection boundaries. To test the full loop, try Supermemory with one recurring workflow and score the user outcome before increasing the amount of information remembered.