Benchmark coverage matrix

Which memory and agent benchmarks each system has reported scores on, and how those scores compare. Memory papers split into two camps: a memory-specific bench family (LongMemEval, LoCoMo, BABILong, ConvoMem, RULER, MemoryAgentBench, NIAH) that tests recall across sessions and long contexts, and a domain-specific family (GAIA, SWE-bench, WebArena, OSWorld, AIME) that tests downstream task performance where memory is one ingredient among many. Coverage matters because a benchmark with only a handful of reporters can't drive head-to-head comparison — and the memory-specific family is genuinely under-adopted relative to its design intent.

119 systems with at least one score · 25 distinct benchmarks tracked · most-reported: LoCoMo (31 systems)

How scores are extracted

Source priority: cells.perf.value first; cells.claims.value only when perf doesn't mention the benchmark. Cells marked as no-data or sentinels like "no public benchmark scores found" are skipped.
Canonicalisation: aliases collapse to one name — LMES, LME, LongMem-Eval, LongMemEval-S all become LongMemEval; SciWorld becomes ScienceWorld; BABI-Long becomes BABILong.
Score-shape priority (clipped at neighbouring benchmark mentions): (1) unit-suffixed in left window (92.2%, +18.7pp, 0.91 F1); (2) unit-suffixed in right window; (3) bare decimal in perf only (0.700 LoCoMo); (4) short bare int adjacent in perf only (30 ConvoMem). Claims-mode skips bare numbers — they're usually pass@k, k-shot, or version-number noise.
Parser self-report: 81% of perf mentions yielded a parseable score (99/122); 9% of claims mentions did (4/47). Cells without a score still get a presence ✓.

click a cell or system name → main table filtered to that record

Well-covered (≥10 systems) 6

LoCoMo 31
LongMemEval 19
MMLU 17
GAIA 13
SWE-bench 12
HotpotQA 10

Emerging (5–9 systems) 5

ALFWorld 9
AIME 8
NIAH 7
WebArena 5
BABILong 5

Too narrow to compare (<5 systems) 14

OSWorld 4
NaturalQuestions 4
BrowseComp 3
MT-Bench 3
LongBench 3
RULER 3
AppWorld 2
ScienceWorld 2
MemoryAgentBench 2
TriviaQA 2
ConvoMem 2
Mind2Web 1
PersonaBench 1
ImplicitMemBench 1

System	LoCoMo 31	LongMemEval 19	MMLU 17	GAIA 13	SWE-bench 12	HotpotQA 10	ALFWorld 9	AIME 8	NIAH 7	WebArena 5	BABILong 5	OSWorld 4	NaturalQuestions 4	BrowseComp 3	N
Anthropic Claude (foundation models) T1			~77%		~80%			~88%							3
OpenAI GPT family (GPT-5 / GPT-4o / o3 / o4) T1			✓		~94%			~86%							3
A-MEM T3	✓	60.6%													2
Agent KB T4				+18.7pp	+4.0pp										2
AgentFold T4				✓										36.2%	2
Agentic Memory T4						+23.52pp	54.31%								2
Agentic Plan Caching (APC) T3				✓				✓							2
Alibaba Qwen 3 family T1			✓					~80%							2
Anthropic Computer Use T1					✓							✓			2
ByteRover T2	92.2%	92.8%													2
DeepAgent T4				✓			✓								2
DeepSeek R1 / V3 family T1			✓					~76%							2
Dynamic Cheatsheet T4			✓					50.0%							2
Engram (DeepSeek) T1			+4.0 poi						84.2						2
ExpeL T3						✓	59%								2
ExpeL T4						✓	✓								2
GAM (General Agentic Memory) T4	✓					64.07 F1									2
Google Gemini 3 family T1			✓					~85%							2
IterResearch T4				✓										✓	2
Letta / MemGPT T1	74.0%	83.2%													2
LiCoMemory T4	40%	73.8%													2
LightMem T4	✓	6.40%													2
LM2 T4			+5.0%								+86.3%				2
LongRAG T4						2							1		2
MAGMA T4	0.700	✓													2
Magnetic-One T3				✓						✓					2
Mem0 T1	91.6	~26%													2
MemAgent T4						✓			✓						2
MemoBrain T4				✓										✓	2
MemOS (MemTensor) T3	-30%	80%													2
Mistral Large 2 / Mixtral family T1			84%		~92%										2
OpenAI Operator T1										38.1%		38			2
SGMem T4	✓	✓													2
Supermemory T1	✓	71.43%													2
TiMem T4	75.30%	76.88%													2
Titans (Google) T4									✓		✓				2
xAI Grok 4 T1			~30%					~80%							2
01.AI Yi family T2			024												1
Agent S T4												20.58%			1
Agent Workflow Memory T3										+51.1%					1
Alita T4				75.15%											1
Alita-G T4				83.03%											1
Amazon Nova family T2			~85%												1
ARMT (Associative RMT) T4											79.9%				1
Atlas T3											+80%				1
ATLAS T4											+80%				1
Backboard T2	✓														1
BAI-LAB MemoryOS T3	+48.36%														1
Beyond RAG for Agent Memory T4	✓														1
BrowserAgent T4						✓									1
CDMem T3							85.8%								1
ChatGPT — Codex (cloud agent) T1					✓										1
Claude Code (Anthropic) T1					~72%										1
Cohere Command R+ / Command A T1			~85%												1
COLA T4				31.89%											1
Darwin Gödel Machine T4					50.0%										1
Diffbot T1			70.36%												1
Differentiable Search Index (DSI) T3													✓		1
EMAT T3													44.3 EM		1
EverMemOS T4		83.0%													1
EVOLVE-MEM T3	58.3%														1
GWM (Gaussian World Models) T3				✓											1
H²R T4							75.9%								1
HeLa-Mem T4	✓														1
HELMET T3									✓						1
Hindsight (Vectorize) T1		91%													1
Inducing Programmatic Skills T4										40.4%					1
KAG T3						+33.5%									1
LangMem (LangChain) T2	58.10%														1
Mastra Memory T2		94.87%													1
MATTER T4													✓		1
MEM1 T4						✓									1
Memento T4				87.88%											1
MemInsight T3	+34%														1
MemLoRA T4	81.3														1
MemMachine T2	0.9169														1
MemMachine (paper) T4	0.9169														1
MemoRAG T3		~62%													1
Memoria (MatrixOrigin) T2		88.78%													1
MemoryArena T4	✓														1
Memp T4							✓								1
MemPalace T2		96.6%													1
MemR³ T4	✓														1
MemRL T4							✓								1
Memvid T2	+35%														1
Meta Llama 4 family T1			~80%												1
Microsoft Phi-4 family T2			84.8%												1
MIRIX T4	85.4%														1
Mistral Vibe (Remote Agents) + Mistral Medium 3.5 T1					77.6%										1
OMEGA T2		95.4%													1
OpenAI Codex CLI T1					✓										1
OpenDevin / OpenHands T2					✓										1
OpenHands (All Hands AI) T1					✓										1
RAFT T3						✓									1
Reka Core / Flash / Edge T2			~83%												1
Retroformer T4							✓								1
RGMem T4	✓														1
RMM (Reflective Memory Management) T3		10%													1
RULER T3									✓						1
SeCom T3	✓														1
ShadowKV (ByteDance) T2									✓						1
SkillWeaver T4										8%					1
SnapKV T3									✓						1
SuperLocalMemory T1	74.8%														1
SYNAPSE T4	2 F1														1
UI-TARS T2												24.6%			1
Wayve GAIA-2 / GAIA-3 T1				-2											1
Zep & Graphiti T1	84%														1

Per-benchmark leaderboards

Top systems on each benchmark, ranked by reported absolute score. Deltas (+18.7pp) are excluded — they aren't comparable across baselines. The tier pill marks the source record's tier (T1 commercial → T5 informal). Scores from claims rather than perf are marked with a small dot.

LongMemEval

8 with absolute score

1 MemPalace T2 96.6%
2 OMEGA T2 95.4%
3 Mastra Memory T2 94.87%
4 ByteRover T2 92.8%
5 Hindsight (Vectorize) T1 91%
6 Memoria (MatrixOrigin) T2 88.78%
7 Letta / MemGPT T1 83.2%
8 EverMemOS T4 83.0%

LoCoMo

8 with absolute score

1 ByteRover T2 92.2%
2 Mem0 T1 91.6
3 MIRIX T4 85.4%
4 Zep & Graphiti T1 84%
5 MemLoRA T4 81.3
6 TiMem T4 75.30%
7 SuperLocalMemory T1 74.8%
8 Letta / MemGPT T1 74.0%

BABILong

1 with absolute score

1 ARMT (Associative RMT) T4 79.9%

ConvoMem

1 with absolute score

1 Mem0 T1 30

GAIA

4 with absolute score

1 Memento T4 87.88%
2 Alita-G T4 83.03%
3 Alita T4 75.15%
4 COLA T4 31.89%

MMLU

8 with absolute score

1 Cohere Command R+ / Command A T1 ~85%
2 Amazon Nova family T2 ~85%
3 Microsoft Phi-4 family T2 84.8%
4 Mistral Large 2 / Mixtral family T1 84%
5 Reka Core / Flash / Edge T2 ~83%
6 Meta Llama 4 family T1 ~80%
7 Anthropic Claude (foundation models) T1 ~77%
8 Diffbot T1 70.36%

SWE-bench

6 with absolute score

1 OpenAI GPT family (GPT-5 / GPT-4o / o3 / o4) T1 ~94%
2 Mistral Large 2 / Mixtral family T1 ~92%
3 Anthropic Claude (foundation models) T1 ~80%
4 Mistral Vibe (Remote Agents) + Mistral Medium 3.5 T1 77.6%
5 Claude Code (Anthropic) T1 ~72%
6 Darwin Gödel Machine T4 50.0%

HotpotQA

2 with absolute score

1 GAM (General Agentic Memory) T4 64.07 F1
2 LongRAG T4 2

ALFWorld

4 with absolute score

1 CDMem T3 85.8%
2 H²R T4 75.9%
3 ExpeL T3 59%
4 Agentic Memory T4 54.31%

AIME

7 with absolute score

1 Anthropic Claude (foundation models) T1 ~88%
2 OpenAI GPT family (GPT-5 / GPT-4o / o3 / o4) T1 ~86%
3 Google Gemini 3 family T1 ~85%
4 Alibaba Qwen 3 family T1 ~80%
5 xAI Grok 4 T1 ~80%
6 DeepSeek R1 / V3 family T1 ~76%
7 Dynamic Cheatsheet T4 50.0%

NIAH

1 with absolute score

1 Engram (DeepSeek) T1 84.2

WebArena

3 with absolute score

OSWorld

3 with absolute score

1 OpenAI Operator T1 38
2 UI-TARS T2 24.6%
3 Agent S T4 20.58%

NaturalQuestions

2 with absolute score

1 EMAT T3 44.3 EM
2 LongRAG T4 1

BrowseComp

1 with absolute score

1 AgentFold T4 36.2%

Memory-specific vs domain-specific adoption

Each system is classified by which benchmark family it reports on. The stacked bar shows the population split; the three columns underneath list who falls into each bucket. So what: the "domain-only" fraction is the evidence for the recurring observation that memory papers default to the agent benchmark of their target domain rather than reporting on a memory-specific axis their architecture would seem to target.

Memory-only · 48

Both · 6

Domain-only · 65

Memory-only

Both families

Domain-only

Most-benchmarked systems

Top-10 systems by distinct benchmarks reported under the current tier filter.

1 Agentic Memory ALFWorldHotpotQAScienceWorld 3
2 Anthropic Claude (foundation models) AIMEMMLUSWE-bench 3
3 GAM (General Agentic Memory) HotpotQALoCoMoRULER 3
4 Mem0 ConvoMemLoCoMoLongMemEval 3
5 MemAgent HotpotQANIAHRULER 3
6 OpenAI GPT family (GPT-5 / GPT-4o / o3 / o4) AIMEMMLUSWE-bench 3
7 ShadowKV (ByteDance) LongBenchNIAHRULER 3
8 Supermemory ConvoMemLoCoMoLongMemEval 3
9 A-MEM LoCoMoLongMemEval 2
10 Agent KB GAIASWE-bench 2

What to make of this

LoCoMo and LongMemEval are the de-facto memory leaderboards. If you want to compare a new memory system head-to-head against shipping work, these are the two benchmarks you'd run first. T2 productized systems (MemPalace, OMEGA, Mastra, ByteRover) crowd the top of LongMemEval; T1 commercial (Mem0, Zep, Letta) cluster around 84-91% on LoCoMo.
ConvoMem is under-adopted. Only ~2 systems report on it (Mem0 plus an opaque Supermemory mention), so it can't drive cross-system comparison — it's effectively a single-paper artefact. It shows up in the "too narrow" tier above.
Memory papers benchmark on their target domain. A paper proposing a memory system for web agents (Mind2Web, WebArena) usually reports on its target domain rather than on a memory-specific axis. The "both families" bucket is small relative to "domain-only" — the user's recurring observation about evaluation discipline in the field.
Filter to T1 only to see the commercial story directly. The commercial vendors converge on LoCoMo + LongMemEval as the comparison axis — that convergence is what makes those two benchmarks credible.