Memory benchmarks
Vendors' memory benchmark numbers aren't comparable: each picks its own answer model, judge, prompt and subset, and several have publicly disputed each other's results. This page keeps two things apart. First, our own runs, where every system gets the same data, answer model, judge and retrieval budget. Second, the scores vendors and papers report, shown with their sources and labelled as not reproduced.
Reproducible runs
No published runs yet. The harness is in the repo and tested offline. The first real run needs API keys, so it's waiting on the maintainer.
- Datasets: LongMemEval and LoCoMo.
- Systems: Mem0, Zep, Graphiti, Letta, Cognee, Supermemory and Honcho, plus full-context, BM25 and embedding-RAG baselines.
- Fixed for every system: answer model, judge model and prompt, top-k retrieval budget, and the same questions.
- Reproduce it yourself: bench/ (
python -m memory_bench.run). Adapter notes are in ADAPTERS.md.
Reported scores (not reproduced) 33
These are copied from vendor pages and papers. Setups differ, so don't rank systems by these numbers. Use them as leads, and check the source for the model, judge and subset used.
| System | Who says so | Claim | Source |
|---|---|---|---|
| Backboard | vendor | ships own LoCoMo benchmark fork (Backboard-Locomo-Benchmark) — score not headlined | github.com |
| ByteRover | vendor | LoCoMo 92.2% (vendor; ByteRover 2.0) | byterover.dev |
| Hindsight (Vectorize) | vendor | LMES 91% (vendor; highest among standalone memory services per VentureBeat post) | venturebeat.com |
| Letta / MemGPT | vendor | LMES 83.2 (vendor); LoCoMo 74.0 with gpt-4o-mini (vendor) | letta.com |
| Mastra Memory | vendor | LMES 94.87% with gpt-5-mini; 84.23% with gpt-4o (vendor) | mastra.ai |
| Mem0 | vendor | LoCoMo 91.6 (vendor); LMES ~26% improvement over OpenAI; ConvoMem 30 (vendor) — disputed by Zep counter-analysis | mem0.ai |
| MemMachine | vendor | 0.9169 LoCoMo F1 (paper Table 4; SOTA at submission) | arxiv.org |
| Memoria (MatrixOrigin) | vendor | LMES 88.78% overall (vendor blog) | medium.com |
| Memvid | vendor | LoCoMo +35% (vendor); sub-millisecond ops 0.025ms P50 | github.com |
| OMEGA | vendor | LMES 95.4% (466/500) at 50ms (vendor) — claimed highest published | omegamax.co |
| Pydantic-AI + Hindsight | vendor | LMES 91% (Hindsight; vendor) | hindsight.vectorize.io |
| SuperLocalMemory | vendor | LoCoMo 74.8% Mode A (zero LLM); 87.7% Mode C (vendor; arxiv 2603.14588) | arxiv.org |
| Supermemory | vendor | LMES self-reported #1 (Multi-Session 71.43% / Temporal 76.69%); LoCoMo #1 self-reported; ConvoMem #1 self-reported (late 2025) ⚠ self-reported | supermemory.ai |
| Zep & Graphiti | vendor | LMES ~94.8% (vendor blog); disputed Mem0 LoCoMo via published counter-analysis | blog.getzep.com |
| A-MEM | paper | LongMemEval: reports 60.6% overall accuracy with GPT-4o-mini (atomic-note + Zettelkasten linking); LoCoMo ablations show consistent gains. | arxiv.org |
| BAI-LAB MemoryOS | paper | +48.36% F1 vs baselines on LoCoMo (EMNLP 2025 paper) | aclanthology.org |
| EverMemOS | paper | 83.0% LMES (arXiv 2601.02163) | arxiv.org |
| EVOLVE-MEM | paper | 58.3% LoCoMo (OpenReview paper) | openreview.net |
| ImplicitMemBench | paper | 64.1% ImplicitMemBench (overall) | arxiv.org |
| LightMem | paper | +2.09–6.40% LongMemEval-S/LoCoMo (GPT) 38x token reduction | arxiv.org |
| MAGMA | paper | 0.700 LoCoMo overall (vs Full-Context 0.481) +18.6–45.5% relative gains | arxiv.org |
| Mem-α | paper | 0.642 MemoryAgentBench avg (vs RAG-Top2 0.567) | arxiv.org |
| MemInsight | paper | +34% ret LoCoMo | aclanthology.org |
| MemLoRA | paper | Outperforms 10x larger baseline models (Gemma2-27B) and matches 60x larger models (GPT-OSS-120B) on LoCoMo; MemLoRA-V scores 81.3 vs 23.7 on multimodal VQA tasks. | arxiv.org |
| MemMachine (paper) | paper | 0.9169 on LoCoMo (gpt-4.1-mini) — above Mem0, Zep, Memobase, LangMem. 93.0% on LongMemEvalS. ~80% fewer input tokens than Mem0 in matched comparisons. | arxiv.org |
| MemoRAG | paper | LongMemEval: ~62% overall accuracy with beacon_ratio=16; up to 1M-token contexts; 30× prefill speed-up via memory beacons. | arxiv.org |
| MemOS (MemTensor) | paper | LongMemEval: 80%+ accuracy with GPT-4o-mini; MemCubes outperform vanilla RAG by 20-30% on LoCoMo (vendor benchmark v2.0). | arxiv.org |
| MemPalace | paper | 96.6% LMES (vendor-self-reported; flagged for verification) | mempalace.net |
| MemR³ | paper | outperfo LoCoMo | arxiv.org |
| MIRIX | paper | 35% higher accuracy than RAG baseline with 99.9% storage reduction on ScreenshotVQA; state-of-the-art 85.4% on LOCOMO. | arxiv.org |
| RMM (Reflective Memory Management) | paper | >10% absolute accuracy gain on the LongMemEval benchmark over the strongest prior retrieval-then-reason baseline; ablations show both prospective and retrospective reflection are necessary. | arxiv.org |
| SYNAPSE | paper | +7.2 F1 over the strongest baseline on the LoCoMo long-conversation memory benchmark; ablations isolate the contribution of synaptic consolidation versus retrieval-only memory. | arxiv.org |
| TiMem | paper | 75.30% LoCoMo 76.88% LMES | arxiv.org |
Related finding: most products publish no peer-reviewed benchmark.