How to Audit Agent Trajectories for Memory Failures

A trajectory audit examines the sequence of retrievals, decisions, and tool calls that led to an agent's result. Its purpose is to find the first observable failure, including cases where a plausible final answer hides missing or misused evidence.

A useful audit starts with recorded evidence and a denominator you can explain. Separate observations from explanations: a missing fact is an observed failure, while an indexing delay is a possible cause to investigate.

Define the unit before counting failures

A run can contain several steps, and one failed step can cause many later symptoms. Record both run-level outcomes and step-level events. Otherwise, a long failing run can dominate the counts while short successful runs barely contribute.

For illustration, 20 runs with five steps each produce 100 steps. Four failed steps across two runs mean a 4% step-failure rate and a 10% run-failure rate. Those describe different quantities; neither establishes how your production workload behaves.

Capture the evidence chain

Each step should identify its run, parent step, task, authorized scope, source IDs, retrieved evidence, selected context, and tool outcome. Record processing readiness and relevant configuration versions. Prefer identifiers and status fields to copying private customer text into a second logging system.

A retrieval response is not the same as the context sent to the model. Preserve both evidence sets so an omitted passage can be traced to retrieval or context assembly. When a tool changes external state, record the observed result separately from the model's intention.

Label the first failure consistently

Start with a small set of labels: source absent, source unready, wrong scope, wrong version, poor ranking, dropped context, misinterpreted evidence, and failed tool action. Leave a case unresolved when the trace cannot distinguish two explanations.

Write one positive and one negative example for each label. Have ambiguous cases reviewed against the same rules. Reclassifying a case after learning more is reasonable; silently changing the rules between baseline and follow-up makes the trend unreliable.

Replay a fix against a stable set

Preserve a redacted replay fixture and compare the affected steps after a change. Keep unaffected cases too: increasing retrieval breadth might fix a missing fact while introducing irrelevant instructions or extra latency elsewhere.

Turn recurring findings into the logs and alerts described in the memory observability guide.

The memory debugging guide helps localize individual failures. Use MemoryBench where its supported evaluation fits, and keep application-specific traces alongside benchmark results. Begin with a handful of inspectable runs before scaling the audit; reliable labels matter more than an impressive sample count.