Evo-Memory
https://arxiv.org/abs/2511.20857
Google streaming benchmark for test-time learning: agents must search, adapt and update memory across task sequences. It implements 10+ memory modules on 10 datasets and introduces the ExpRAG and ReMem baselines. Traction: Google (Ed Chi, Fernando Pereira et al.) authorship
Active · Verified 2026-10-23 · fresh as of 2026-10-23 · 7 cited facts, 0 with a third-party source · 1 recorded change · Suggest a correction
At a glance
- Tier
- T3
- Section
- Memory benchmarks & evaluation
- Created
- searched not found fix
- Latest release
- searched not found fix
- GitHub
- not applicable — no GitHub repo fix
- Pricing
- searched not found fix
- Funding
- searched not found fix
- Freshness
- 2026-10-23 fix
- Layer
- agent-infra repo fix
Memory taxonomy
- storage
- none-trivial
- retrieval
- none
- persistence
- cross-session
- update
- read-only
- unit
- trajectory
- governance
- n/a
- conflict
- out-of-scope
When to use
Optimised for: searched not found
Anti-fit: searched not found
Pros & cons
Pros
searched not found
Cons
searched not found
Claims & capabilities
Abstract: Statefulness is essential for large language model (LLM) agents to perform long-term planning and problem-solving. This makes memory a critical component, yet its management and evolution remain largely underexplored. Existing evaluations mostly focus on static conversational settings, where memory is passively retrieved from dialogue to answer queries, overlooking the dynamic ability to accumulate and reuse experience across evolving task streams. In real-world environments such as interactive problem assistants or embodied agents, LLMs are required to handle continuous task streams, yet often fail to learn from accumulated interactions, losing valuable contextual insights, a limitation that calls for test-time evolution, where LLMs retrieve, integrate, and update memory continuously during deployment. To bridge this gap, we introduce Evo-Memory, a comprehensive streaming benchmark and framework for evaluating self-evolving memory in LLM agents. Evo-Memory structures datasets into sequential task streams, requiring LLMs to search, adapt, and evolve memory after each interaction. We unify and implement over ten representative memory modules and evaluate them across 10 div
Technical surface
- API surface
- searched not found
- Backend storage
- searched not found
- Deployment
- searched not found
- Embedding model
- searched not found
- Multi-tenancy
- searched not found
- MCP
- searched not found
- A2A
- searched not found
- OpenTelemetry
- searched not found
Similar systems
Other memory benchmarks & evaluation in the catalog, ranked by inbound references.
- LoCoMo T3
Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.
- GAIA T3
Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset
- LongBench T3
Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.
- LongMemEval T3
5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).
- RULER T3
NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.
- InfiniteBench (∞Bench) T3
Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.
Sources (6 cited facts, 0 third-party)
Each fact links to the page it came from; the badge says who published it (how to read sources).
- What it is Benchmark vendor
- What it does & what's distinct Google streaming benchmark for test-time learning: agents must search, adapt and update … vendor
- Claims / benefits Abstract: Statefulness is essential for large language model (LLM) agents to perform lon… vendor
- License unknown vendor
- Project & sources https://arxiv.org/abs/2511.20857 ; https://arxiv.org/abs/2511.20857v1 vendor
- taxonomy-three-layer-rollup agent-infra repo
What changed
Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)
- 2026-10-23 Added to the catalog #222