Research

Delivered, Not Used: Evaluating Agent Memory at Its Point of Use

Changran Xu, Zeju Li, Qiang Xu The Chinese University of Hong Kong

Read the paper abstract

Task success, retrieval quality, and cost do not show whether an LLM agent used a delivered memory or what followed from that use. We introduce MemUse, which evaluates memory at its point of use in recorded trajectories: each memory delivery is compared with what the agent could already see and followed through the agent's later actions and their consequences. An agentic judge rates each delivered entry's relevance, accuracy, novelty, influence, and observed effect; programmatic checks enforce complete reading of cited evidence, and fixed rules aggregate the ratings into verdicts linked to memory histories and delivery cost. On 8,000 SWE-bench Verified episodes from three memory systems, resolve rates stay within 0.4 percentage points of a no-memory baseline, yet MemUse shows that most delivered entries were irrelevant, most relevant entries went unused, and in two systems most deliveries restated content the agent could still see. It also surfaces effects that outcomes hide: 23.2% of episodes with beneficial memory use still failed, and 5 of 16 with harmful use still succeeded.

Key findings

The study uses 500 SWE-bench Verified tasks, three memory systems in four configurations, and four runs per configuration. The 8,000 memory episodes are compared with 2,000 no-memory episodes.

8,000Memory-enabled episodes across four configurations
≤ 0.4 ppLargest absolute resolve-rate difference from baseline (percentage points)
23.2%Beneficial-use episodes that still failed
5 of 16Harmful-use episodes that still passed

The beneficial/harmful-use outcome figures include only episodes with recorded harness results.

Outcomes hide memory effects

Follow the recorded actions in two episodes. Select a marked step to inspect the evidence.

A TencentDB skill sends the agent into the wrong component at steps 2–5. The agent finds the real fix at step 9, and the task passes.

A MemGovern card helps the agent add a subset handler at step 8. Another error remains unfixed, and the task fails.

Memory effects are judge assessments. Task outcomes come from the SWE-bench harness. Step numbers are local to each episode.

Memory profiles

The six axes summarize entry shares: relevance, corroboration, information gain, acted on, benefit, and harm. Each profiled span has equal weight in the plotted means. See how profiles differ from verdicts.

Compare verdicts, pass rates, and cost in the Results explorer.

Scope of these results

These findings describe the studied configurations on SWE-bench Verified. The paper supplied gold patches to the judge but withheld harness outcomes. Recorded attribution is an observational assessment; it does not establish a memory system’s causal effect on task success or its performance on other tasks.

Citation

If MemUse, memory-bridge, or traj-recorder helps your research, please cite the paper.

BibTeX
@misc{xu2026deliveredusedevaluatingagent,
      title={Delivered, Not Used: Evaluating Agent Memory at Its Point of Use},
      author={Changran Xu and Zeju Li and Qiang Xu},
      year={2026},
      eprint={2611.01718},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2611.01718},
}