Evaluate memory
beyond task success.
MemUse traces delivered memory through recorded agent actions. See which memories helped, hindered, or went unused.
A detour hidden by success.
- Memory delivered
A skill suggests a fix for a different bug.
- Agent action Steps 2–5
The agent follows it into the wrong component.
- Later action Step 9
The agent finds the real fix and completes the task.
Useful guidance in a failed task.
- Memory delivered
A memory card points to a needed subset handler.
- Agent action Step 8
The agent uses the guidance to add the correct handler.
- Remaining error
Another error remains unfixed, so the task fails.
Pass rate is only
part of the story.
Pass rate measures the agent and memory system together. It does not show which delivered memories helped or hindered the agent.
MemUse assesses how delivered memory was used and what followed from that use.
Explore the resultsMemory graph
memgraphHandle empty inputs before the batch loop.
Evaluation
injudgeRecall M1 · Task 02
Memory action
The agent retrieves a memory before it edits the code.
- Delivered memoryThe text the agent received
- Later agent actionThe code change after delivery
- Observed consequenceThe result of the following check
- Relevance
- Relevant
- Accuracy
- Corroborated
- Novelty
- Novel
- Influence
- Acted on
- Effect
- Improved
Select a step or a recall to inspect it.
Connect. Record. Evaluate.
To create new annotated runs with this workflow, you need memory-bridge and traj-recorder. To evaluate existing compatible recordings, install only MemUse.
memory-bridge
Connect your agent harness to a memory system. Mark memory operations and task boundaries.
02traj-recorder
Capture model calls and memory annotations in a recording that MemUse can read.
03MemUse
Inspect the graph with memgraph. Judge memory use and read the evidence with injudge.
Delivered,
Not Used.
Evaluating Agent Memory at Its Point of Use
Most delivered entries were irrelevant. Most relevant entries went unused. The paper shows where useful guidance and harmful detours appeared in the recorded runs.
Memory-enabled episodes
on SWE-bench Verified
Largest absolute resolve-rate difference
from the no-memory baseline
Three memory systems. Four configurations.
Findings apply to the studied SWE-bench configurations.
See how memory shapes
your agent’s actions.
Inspect a memory graph, prepare the evidence, then judge a small sample.
Open the quickstartWork in progress
Help shape what comes next.
Contributions are welcome. Report a bug, improve the docs, or connect a new harness, memory system, or benchmark. Interfaces and formats can change.
How to contribute