EvoMemBench
https://www.alphaxiv.org/abs/2605.18421
Benchmark that crosses memory scope (in-episode vs cross-episode) with content (knowledge vs execution) and compares 15 memory methods with long-context baselines under one protocol. Traction: Standardized comparison of 15 memory methods; HKUST(GZ) and others
Active · Verified 2026-10-23 · fresh as of 2026-10-23 · 12 cited facts, 0 with a third-party source · 1 recorded change · Suggest a correction
At a glance
- Tier
- T3
- Section
- Memory benchmarks & evaluation
- Created
- 2026-05-18 repo fix
- Latest release
- last commit 2026-07-03 repo fix
- License
- unspecified vendor fix
- GitHub
- https://github.com/DSAIL-Memory/EvoMemBench (14★) repo fix
- Pricing
- searched not found fix
- Funding
- searched not found fix
- Freshness
- 2026-10-23 fix
- Layer
- agent-infra repo fix
- Maintainer health
- 1 issues in 90 days · 0 releases in 12 months · bus factor 1 — as of 2026-10-31 repo
Memory taxonomy
- storage
- none-trivial
- retrieval
- none
- persistence
- cross-session
- update
- read-only
- unit
- episode
- governance
- n/a
- conflict
- out-of-scope
When to use
Optimised for: searched not found
Anti-fit: searched not found
Pros & cons
Pros
searched not found
Cons
searched not found
Claims & capabilities
Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agents, as it enables them to store, update, and retrieve information over time. This ability remains under-evaluated, largely because existing benchmarks do not provide a systematic way to assess memory mechanisms. In this paper, we study agent memory from a self-evolving perspective and introduce EvoMemBench, a unified benchmark organized along two axes: memory scope (in-episode vs. cross-episode) and memory content (knowledge-oriented vs. executio
Technical surface
- API surface
- searched not found
- Backend storage
- searched not found
- Deployment
- searched not found
- Embedding model
- searched not found
- Multi-tenancy
- searched not found
- MCP
- searched not found
- A2A
- searched not found
- OpenTelemetry
- searched not found
Similar systems
Other memory benchmarks & evaluation in the catalog, ranked by inbound references.
- LoCoMo T3
Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.
- GAIA T3
Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset
- LongBench T3
Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.
- LongMemEval T3
5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).
- RULER T3
NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.
- InfiniteBench (∞Bench) T3
Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.
Sources (11 cited facts, 0 third-party)
Each fact links to the page it came from; the badge says who published it (how to read sources).
- What it is Benchmark vendor
- What it does & what's distinct Benchmark that crosses memory scope (in-episode vs cross-episode) with content (knowledg… vendor
- Claims / benefits Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, plann… vendor
- Created 2026-05-18 repo
- Latest release last commit 2026-07-03 repo
- License unspecified vendor
- GitHub https://github.com/DSAIL-Memory/EvoMemBench (14★) repo
- Mindshare 14 GitHub stars repo
- Project & sources https://www.alphaxiv.org/abs/2605.18421 ; https://github.com/DSAIL-Memory/EvoMemBench vendor
- cost-batching yes repo
- taxonomy-three-layer-rollup agent-infra repo
What changed
Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)
- 2026-10-23 Added to the catalog #222