Quickstart
Turn an annotated recording into a memory graph and a judge report. Start with an existing recording, or create the sample below.
No recording yet? Install MemUse in step 1, then follow Create a sample recording before step 2.
Before you begin
| Task | Requirements |
|---|---|
| Inspect a recording offline | Git, uv, Python 3.11+, and schema-10 recordings with memory_* annotations. |
| Run the judge | Node.js 22.19+, npm, and a judge endpoint/key. |
| Create the optional sample | Docker, an agent model endpoint/key, and a mem0 Platform account/key. The recipe installs memory-bridge and traj-recorder. |
Keep the complete recording directory, including its call captures. A plain model transcript cannot reconstruct memory annotations.
Cloning and dependency installation need network access. Graph rendering and evidence preparation make no model calls. Recording a live task and running the judge can incur charges.
1. Install MemUse
Run from the directory where you keep your checkouts:
git clone --branch agentic_search https://github.com/ChangranXU/mem-use.git mem-use
cd mem-use
uv syncAll evaluation commands below run from the mem-use folder.
2. Inspect the memory graph
Replace the example path with your recorder directory, a trajectory.jsonl file, or a directory of recordings:
REC=/absolute/path/to/recording
uv run -m memgraph "$REC" -o memgraph.html --json memgraph.jsonOpen memgraph.html in a browser. Check the validation notes, then select a memory to trace its creation, updates, recalls, and deletion. Upward triangles mark writes; downward triangles mark recalls.
Choose the right input for a resumed run
When the runner provides manifests, use evaluation.json for checkpoint-credited history. Use evaluation-attempts.json for all attempts, or evaluation-discarded.json for interrupted or uncheckpointed work. Load one view at a time; do not combine it with a recursive scan of the same recordings.
See the input guide for supported views.
3. Check the judge input offline
uv run -m injudge "$REC" --dry-run --limit 3
uv run -m injudge "$REC" --prepare-only -o output/prepared/--dry-run prints evidence for up to three eligible spans and writes no files. --prepare-only saves episode evidence under output/prepared/<run-stamp>/. Neither needs Node.js or judge credentials.
Review diagnostics before paying for a judgment. Missing episode boundaries or gaps in a delivery’s consuming input can block that episode. The method guide defines episodes, entries, and exposure.
4. Judge a small sample
Install the judge dependencies and copy its configuration template. Keep an existing configured .env file:
npm --prefix agent install
test -f .env || cp .env.example .envEdit these values in .env:
MODEL="your-judge-model"
API_KEY="your-judge-api-key"
BASE_URL="https://your-endpoint.example"
API="openai-chat"For an OpenAI-compatible endpoint, use the base URL that serves /chat/completions. For Anthropic, set API="anthropic" and use its API host root. See configuration for budgets and concurrency.
uv run -m injudge "$REC" --limit 3 --html -o output/Review the sample before removing --limit 3. To select episodes, add --episode TASK_ID with an exact recorded ID; the flag is repeatable.
No reference answer or task score is required. To attach them, use the gold and outcome guide.
5. Inspect one judgment
Each judge run creates a new output/<run-stamp>/ folder:
| Output | Use |
|---|---|
summary.md | Start with the counts, coverage, diagnostics, and suggested spans to review. |
memgraph-verdicts.html | Open a recall to inspect its entry signals, verdict, and cited steps. |
injudge-report.json / summary.json | Read the full record or compact statistics programmatically. |
evidence/ / agent-runs/ | Audit the files the judge could read and its recorded conversation. |
Choose a non-neutral span, read its analysis, and follow the cited steps. Check unknown values, errors, and unjudged spans before comparing runs. Use verdict rules and entry-share profiles to interpret the result.
Render an existing report again
Use the same recording view and replace RUN with the report folder:
uv run -m memgraph "$REC" \
--verdicts output/RUN/injudge-report.json -o memgraph-verdicts.htmlCreate a sample recording
This optional path runs MiniSWE with mem0 Platform on a small SWE-bench sample. It needs Docker and separate credentials for the agent and mem0. For other products or task suites, use the runner catalog.
Install the recording projects
Start from the parent directory containing your mem-use checkout. If your terminal is still in mem-use after step 1, run cd .. first.
mkdir -p extension
git clone --branch v2 https://github.com/ChangranXU/memory-bridge.git
git clone --branch memory https://github.com/ChangranXU/traj-recorder.git extension/traj-recorder
cd memory-bridge
git clone https://github.com/SWE-agent/mini-swe-agent.git mini-swe-agent
git -C mini-swe-agent checkout 04d809ceab9df28f9adaed044884180159172930
uv syncThe runner starts the recorder from the sibling checkout and resolves its dependencies. Keep running the following commands from memory-bridge.
Configure and check
test -f settings/shared/llm.env || cp settings/shared/llm.env.example settings/shared/llm.env
test -f settings/recorder/roles.env || cp settings/recorder/roles.env.example settings/recorder/roles.env
test -f settings/memory/mem0.env || cp settings/memory/mem0.env.example settings/memory/mem0.envFill in the agent endpoint, model, and key in settings/shared/llm.env. Set MEM0_OPTION=cloud and MEM0_API_KEY in settings/memory/mem0.env. Check the applicable model card and required settings in the mem0 setup guide. Keep existing settings files if already configured.
uv run python scripts/check_settings.py --provider mem0 --mem0-option cloudThis check starts no task or paid model call. For a credentials-free API exercise, use the offline bridge example.
Run and verify
uv run python scripts/swebench/swe-mini-agent/run.py \
--ids scripts/swebench/swe-mini-agent/ids.smoke.txt \
--output runs/mem0-smoke --provider mem0 --mem0-option cloud --strictUse a new output directory. Inspect runs/mem0-smoke/verify.log and summary.json for trace and memory failures. A completed task alone does not verify memory use.
Return to mem-use with cd ../mem-use. Continue at step 2, setting REC=../memory-bridge/runs/mem0-smoke/recorder.
Record without memory-bridge
traj-recorder can record compatible model traffic on its own. Your client must use the proxy endpoint, and a host must supply annotations for memory analysis. Follow the standalone recorder quickstart.
Common problems
The memory graph is empty
Check that you selected the intended recording and that it contains memory_* events. A none baseline or standalone unannotated proxy recording correctly has no memory operations.
A recall shows “created earlier”
The loaded recordings contain a recall but no observed creation for that item. Load earlier runs if you need that history.
Spans are unassessable
No observed agent step can be attributed as the delivery’s consumer. This is missing consumption evidence, not a judgment that the agent ignored the memory.
injudge exits with status 2
Read the console diagnostics. A normal judge run can also report span errors in summary.md. Offline preparation writes diagnostics to its evidence package; dry-run creates no report.