Evaluate memory
beyond task success.

MemUse traces delivered memory through recorded agent actions. See which memories helped, hindered, or went unused.

One task. Two different signals.

A detour hidden by success.

  1. Memory delivered

    A skill suggests a fix for a different bug.

  2. Agent action Steps 2–5

    The agent follows it into the wrong component.

  3. Later action Step 9

    The agent finds the real fix and completes the task.

Memory use↘ Harmful
Task result✓ Passed

Useful guidance in a failed task.

  1. Memory delivered

    A memory card points to a needed subset handler.

  2. Agent action Step 8

    The agent uses the guidance to add the correct handler.

  3. Remaining error

    Another error remains unfixed, so the task fails.

Memory use↗ Beneficial
Task result× Failed

Pass rate is only
part of the story.

Pass rate measures the agent and memory system together. It does not show which delivered memories helped or hindered the agent.

MemUse assesses how delivered memory was used and what followed from that use.

Explore the results

Memory graph

memgraph
Task 01 Write M1
Task 02 Action Check Credit windowBeneficial use
Task 03 Action Credit windowNot used
Input validation

Handle empty inputs before the batch loop.

Recorded trajectory
memory_writeM1 stored
memory_deliveryM1 → Task 02
agent_actionAdd input guard

Evaluation

injudge

Recall M1 · Task 02

Memory action

The agent retrieves a memory before it edits the code.

  • Delivered memoryThe text the agent received
  • Later agent actionThe code change after delivery
  • Observed consequenceThe result of the following check
Relevance
Relevant
Accuracy
Corroborated
Novelty
Novel
Influence
Acted on
Effect
Improved

Select a step or a recall to inspect it.

Connect. Record. Evaluate.

To create new annotated runs with this workflow, you need memory-bridge and traj-recorder. To evaluate existing compatible recordings, install only MemUse.

Your own harness, memory system, or benchmark?Follow an integration guide

Delivered,
Not Used.

Evaluating Agent Memory at Its Point of Use

Most delivered entries were irrelevant. Most relevant entries went unused. The paper shows where useful guidance and harmful detours appeared in the recorded runs.

8,000

Memory-enabled episodes
on SWE-bench Verified

≤ 0.4 pp

Largest absolute resolve-rate difference
from the no-memory baseline

Three memory systems. Four configurations.
Findings apply to the studied SWE-bench configurations.

See how memory shapes
your agent’s actions.

Inspect a memory graph, prepare the evidence, then judge a small sample.

Open the quickstart

Work in progress

Help shape what comes next.

Contributions are welcome. Report a bug, improve the docs, or connect a new harness, memory system, or benchmark. Interfaces and formats can change.

How to contribute