LoCoMo
https://arxiv.org/abs/2402.17753
Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.
Active · Verified 2026-10-23 · fresh as of 2026-10-23 · 30 cited facts, 6 with a third-party source · Suggest a correction
At a glance
- Type
- Very long-term multi-session dialogue memory vendor fix
- Tier
- T3
- Section
- Memory benchmarks & evaluation
- Latest release
- no-release no-release vendor fix
- License
- MIT (paper code release) vendor fix
- GitHub
- 39★ Jupyter Notebook repo fix
- Pricing
- not applicable — open benchmark; free to use third-party fix
- Funding
- not applicable — not commercial fix
- Freshness
- 2026-10-23 fix
- Layer
- agent-infra repo fix
- Licensing
- open-source vendor fix
- Maintainer health
- 0 issues in 90 days · 0 releases in 12 months · bus factor 1 — as of 2026-10-31 repo
Memory taxonomy
- storage
- n/a
- retrieval
- n/a
- persistence
- n/a
- update
- read-only
- unit
- n/a
- governance
- n/a
- conflict
- n/a
When to use
Optimised for: not applicable - eval dataset, not a system
Anti-fit: not applicable - eval dataset, not a system
Pros & cons
Pros
Long-conversation benchmark — multi-session continuity, temporal reasoning.
Cons
Limited domain coverage; commercial vendors have publicly disputed certain LoCoMo results.
Claims & capabilities
ACL 2024. Used by MemoryOS, MemGPT, A-MEM evaluations. Snap Research / UNC origin.
Technical surface
- API surface
- not applicable — eval dataset, not a system
- Backend storage
- not applicable — eval dataset, not a system
- Deployment
- not applicable — evaluation harness; run locally
- Embedding model
- not applicable — eval dataset, not a system
- Multi-tenancy
- not applicable — eval dataset, not a system
- MCP
- not applicable — benchmark / evaluation harness
- A2A
- not applicable — benchmark / evaluation harness
- OpenTelemetry
- not applicable — benchmark / evaluation harness
Similar systems
Other memory benchmarks & evaluation in the catalog, ranked by inbound references.
- GAIA T3
Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset
- LongBench T3
Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.
- LongMemEval T3
5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).
- RULER T3
NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.
- InfiniteBench (∞Bench) T3
Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.
- Mind2Web T3
Dataset and benchmark for generalist web agents with 2,350 language-instruction tasks on 137 real websites across 31 domains.
Sources (29 cited facts, 6 third-party)
Each fact links to the page it came from; the badge says who published it (how to read sources).
- What it is Very long-term multi-session dialogue memory vendor
- What it does & what's distinct Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent … vendor
- Claims / benefits ACL 2024. Used by MemoryOS, MemGPT, A-MEM evaluations. Snap Research / UNC origin. vendor
- Modalities text + multimodal dialogue (includes image-grounded turns) vendor
- Created 2025-01 vendor
- Latest release no-release no-release vendor
- License MIT (paper code release) vendor
- GitHub 39★ Jupyter Notebook repo
- Mindshare 395 citations (Semantic Scholar); heavily used as industry benchmark by Mem0 Zep Letta B… paper
- Citations 395 cites 197/yr paper
- Pricing not applicable — open benchmark; free to use third-party
- Deployment not applicable — evaluation harness; run locally repo
- Founders / pedigree University of North Carolina Chapel Hill + USC + Snap Inc.; Adyasha Maharana, Dong-Ho Le… vendor
- Reproducibility Disputed ⚠ repo
- Code/weights release Code public + weights closed repo
- Versioning none vendor
- Schema evolution not applicable — fixed evaluation dataset vendor
- Pros Long-conversation benchmark — multi-session continuity, temporal reasoning. vendor
- Cons Limited domain coverage; commercial vendors have publicly disputed certain LoCoMo results. vendor
- Project & sources arxiv 2402.17753 project page vendor
- commit-trajectory [{"ym":"2025-01","cum":9},{"ym":"2025-05","cum":20},{"ym":"2025-07","cum":22}] repo
- citation-trajectory [{"year":2024,"cum":2,"infl":1},{"year":2025,"cum":27,"infl":7},{"year":2026,"cum":38,"i… paper
- cost-input-usd-per-mtok 0 third-party
- cost-output-usd-per-mtok 0 third-party
- cost-tier free third-party
- cost-pricing-model free-tier third-party
- cost-last-verified 2026-06-29 third-party
- licensing-bucket open-source vendor
- taxonomy-three-layer-rollup agent-infra repo
Related systems
References (1)
- Generative Agents cites — S2 isInfluential citation
Referenced by (9)
- A-MEM cites — S2 isInfluential citation
- LiCoMemory cites — S2 isInfluential citation
- LightMem cites — S2 isInfluential citation
- LongMemEval cites — S2 isInfluential citation
- MAGMA cites — S2 isInfluential citation
- MemInsight cites — S2 isInfluential citation
- MemLoRA cites — S2 isInfluential citation
- MemMachine (paper) cites — S2 isInfluential citation
- MIRIX cites — S2 isInfluential citation