Delivered, Not Used: Evaluating Agent Memory at Its Point of Use
Changran Xu, Zeju Li, Qiang Xu The Chinese University of Hong Kong
Read the paper abstract
Task success, retrieval quality, and cost do not show whether an LLM agent used a delivered memory or what followed from that use. We introduce MemUse, which evaluates memory at its point of use in recorded trajectories: each memory delivery is compared with what the agent could already see and followed through the agent's later actions and their consequences. An agentic judge rates each delivered entry's relevance, accuracy, novelty, influence, and observed effect; programmatic checks enforce complete reading of cited evidence, and fixed rules aggregate the ratings into verdicts linked to memory histories and delivery cost. On 8,000 SWE-bench Verified episodes from three memory systems, resolve rates stay within 0.4 percentage points of a no-memory baseline, yet MemUse shows that most delivered entries were irrelevant, most relevant entries went unused, and in two systems most deliveries restated content the agent could still see. It also surfaces effects that outcomes hide: 23.2% of episodes with beneficial memory use still failed, and 5 of 16 with harmful use still succeeded.
Key findings
The study uses 500 SWE-bench Verified tasks, three memory systems in four configurations, and four runs per configuration. The 8,000 memory episodes are compared with 2,000 no-memory episodes.
The beneficial/harmful-use outcome figures include only episodes with recorded harness results.
- Most delivered entries were not relevant to the task.
- Most relevant entries were not used by the agent.
- In two systems, most deliveries repeated text that the agent could still see.
- Each pair of memory systems differs at least tenfold in relevance share or beneficial entries per delivered token. Task resolve rates alone barely separate them.
Outcomes hide memory effects
Follow the recorded actions in two episodes. Select a marked step to inspect the evidence.
A TencentDB skill sends the agent into the wrong component at steps 2–5. The agent finds the real fix at step 9, and the task passes.
A MemGovern card helps the agent add a subset handler at step 8. Another error remains unfixed, and the task fails.
Memory effects are judge assessments. Task outcomes come from the SWE-bench harness. Step numbers are local to each episode.
Memory profiles
The six axes summarize entry shares: relevance, corroboration, information gain, acted on, benefit, and harm. Each profiled span has equal weight in the plotted means. See how profiles differ from verdicts.
Select a system to highlight it, or a point to compare that axis.
Logarithmic scale · 0.001% at the center, 100% at the edge. Lower harm is better.
View exact values
Compare verdicts, pass rates, and cost in the Results explorer.
Scope of these results
These findings describe the studied configurations on SWE-bench Verified. The paper supplied gold patches to the judge but withheld harness outcomes. Recorded attribution is an observational assessment; it does not establish a memory system’s causal effect on task success or its performance on other tasks.
Citation
If MemUse, memory-bridge, or traj-recorder helps your research, please cite the paper.
@misc{xu2026deliveredusedevaluatingagent,
title={Delivered, Not Used: Evaluating Agent Memory at Its Point of Use},
author={Changran Xu and Zeju Li and Qiang Xu},
year={2026},
eprint={2611.01718},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2611.01718},
}