111
Peer-reviewed
66% of mentions
Benchmark integrity
The companion to benchmark coverage. That view answers who benchmarks on what. This view answers the next question: are the numbers trustworthy? Every benchmark mention in the catalog is classified by where the citation lives — peer-reviewed venue, neutral third party, the vendor’s own site — and cross-checked against contradicting claims from other catalog rows. The canonical disputed case (Mem0 vs Zep on LoCoMo) shows up directly in the LoCoMo row of the per-benchmark breakdown.
169 classified benchmark mentions across the 894-record catalog · 25 distinct benchmarks tracked.
2
Independently verified
1% of mentions
29
Vendor-claimed
17% of mentions
27
Disputed
16% of mentions
0
Unverifiable
0% of mentions
Per-benchmark breakdown
For each benchmark, how many mentions fall into each integrity bucket. The cells are clickable — drill into a row to see which records are in that cell. LoCoMo is the canonical example: vendor-claimed scores from commercial vendors, peer-reviewed scores from research papers, and one disputed mention from the Mem0 ↔ Zep rebuttal.
| Benchmark | Total | Peer-reviewed | Independent | Vendor-claimed | Disputed | Unverifiable |
|---|---|---|---|---|---|---|
| LoCoMo | 31 | — | ||||
| LongMemEval | 19 | — | — | |||
| MMLU | 17 | — | — | — | ||
| GAIA | 13 | — | — | — | ||
| SWE-bench | 12 | — | — | |||
| HotpotQA | 10 | — | — | — | — | |
| ALFWorld | 9 | — | — | — | — | |
| AIME | 8 | — | — | — | ||
| NIAH | 7 | — | — | — | — | |
| BABILong | 5 | — | — | — | — | |
| WebArena | 5 | — | — | |||
| NaturalQuestions | 4 | — | — | — | — | |
| OSWorld | 4 | — | — | |||
| BrowseComp | 3 | — | — | — | — | |
| LongBench | 3 | — | — | — | — | |
| MT-Bench | 3 | — | — | — | — | |
| RULER | 3 | — | — | — | — | |
| AppWorld | 2 | — | — | — | — | |
| ConvoMem | 2 | — | — | — | ||
| MemoryAgentBench | 2 | — | — | — | ||
| ScienceWorld | 2 | — | — | — | — | |
| TriviaQA | 2 | — | — | — | — | |
| ImplicitMemBench | 1 | — | — | — | — | |
| Mind2Web | 1 | — | — | — | — | |
| PersonaBench | 1 | — | — | — | — |
Leaderboard of validated winners
Top peer-reviewed scores per benchmark — vendor-claimed numbers are excluded entirely, so this is the leaderboard you can hand to a reviewer with the least caveat. If a benchmark has no peer-reviewed scores under the current filter it doesn’t appear here — by design.
LongMemEval
5 peer-reviewed- 1 EverMemOS T4 83.0% arxiv.org
- 2 MemOS (MemTensor) T3 80% arxiv.org
- 3 TiMem T4 76.88% arxiv.org
- 4 LiCoMemory T4 73.8% arxiv.org
- 5 MemoRAG T3 ~62% arxiv.org
OSWorld
2 peer-reviewedBrowseComp
1 peer-reviewed- 1 AgentFold T4 36.2% arxiv.org
ALFWorld
4 peer-reviewed- 1 CDMem T3 85.8% aclanthology.org
- 2 H²R T4 75.9% arxiv.org
- 3 ExpeL T3 59% arxiv.org
- 4 Agentic Memory T4 54.31% arxiv.org
PersonaBench
1 peer-reviewed- 1 AI PERSONA T4 8.29 arxiv.org
GAIA
4 peer-reviewedBABILong
1 peer-reviewed- 1 ARMT (Associative RMT) T4 79.9% arxiv.org
ScienceWorld
1 peer-reviewed- 1 CDMem T3 56.0% aclanthology.org
SWE-bench
1 peer-reviewed- 1 Darwin Gödel Machine T4 50.0% arxiv.org
AIME
1 peer-reviewed- 1 Dynamic Cheatsheet T4 50.0% arxiv.org
NaturalQuestions
2 peer-reviewedNIAH
1 peer-reviewed- 1 Engram (DeepSeek) T1 84.2 arxiv.org
LoCoMo
5 peer-reviewed- 1 MIRIX T4 85.4% arxiv.org
- 2 MemLoRA T4 81.3 arxiv.org
- 3 TiMem T4 75.30% arxiv.org
- 4 EVOLVE-MEM T3 58.3% openreview.net
- 5 LiCoMemory T4 40% arxiv.org
RULER
2 peer-reviewed- 1 MemAgent T4 95% arxiv.org
- 2 GAM (General Agentic Memory) T4 90% arxiv.org
HotpotQA
2 peer-reviewed- 1 GAM (General Agentic Memory) T4 64.07 F1 arxiv.org
- 2 LongRAG T4 2 arxiv.org
ImplicitMemBench
1 peer-reviewed- 1 ImplicitMemBench T3 64.1% arxiv.org
WebArena
2 peer-reviewed- 1 Inducing Programmatic Skills T4 40.4% arxiv.org
- 2 SkillWeaver T4 8% arxiv.org
TriviaQA
1 peer-reviewed- 1 MATTER T4 56.0 EM arxiv.org
MemoryAgentBench
1 peer-reviewed- 1 Mem-α T4 0.642 arxiv.org
Gaming patterns
Heuristic flags surfaced from the prose of each cell. These don’t automatically demote a row to disputed — they’re shape-of-evidence signals that point at where vendors are benchmarking on terrain that favours them.
Vendor self-defined benchmark
0Benchmark reported by exactly one record, and the citation is on that vendor’s own domain. No external corroboration exists.
No matches under the current filter.
Weak-baseline comparison
8The only comparison in the cell is to a notoriously weak baseline (Full-Context, GPT-3.5, Llama-3.0, "vanilla", or "no-memory baseline").
- AgentEvolver AppWorld T4
- Inducing Programmatic Skills WebArena T4
- LiCoMemory LongMemEval T4
- LiCoMemory LoCoMo T4
- MAGMA LoCoMo T4
- MAGMA LongMemEval T4
- MemOS (MemTensor) LongMemEval T3
- MemOS (MemTensor) LoCoMo T3
Single sub-task only
1The cell explicitly mentions a single sub-task ("single-fact", "sub-task", "only on X") and does not also report an aggregate / overall score.
- ARMT (Associative RMT) BABILong T4