Memory benchmarks & evaluation
46 systems in the memory benchmarks & evaluation category of the AI Agent Infrastructure Landscape, grouped by maturity tier.
Tier 2 — production-ready (4)
- Claude Plays Pokemon (Anthropic) Anthropic agent demo — Claude 3.7/4 Sonnet plays Pokemon Red on Twitch
Anthropic's public Twitch livestream of Claude (3.7 Sonnet → Claude 4 Opus) playing Pokemon Red. Agent uses screenshot vision + Game Boy controller actions + a scratchpad memory file the model can edit. Demonstrates long-horizon agentic …
- ICLR 2026 MemAgents Workshop Workshop venue — RL + cognitive psychology + LLM memory
Full-day hybrid workshop held April 27, 2026, Rio de Janeiro. Scope: episodic and semantic memory, working memory, parametric knowledge, KGs, vector DBs, retrieval pipelines, context management, long-context utilization, temporal credit …
- MineRL Diamond Challenge Minecraft RL benchmark — obtain a diamond
MineRL is a research project / competition series using Minecraft as a long-horizon RL environment. The flagship Diamond Challenge tasks an agent with obtaining a diamond from raw world — requires deep tech-tree planning across thousands…
- NetHack Learning Environment (NLE) Roguelike RL benchmark — NetHack as agent environment
DeepMind / FAIR's NetHack Learning Environment — a Gym-compatible interface to NetHack, the canonical roguelike. Massive procedural state space, permadeath, deep item / skill interactions, ASCII partial-observability. Released at NeurIPS…
Tier 3 — emerging (36)
- AgentBench Benchmark
Multi-environment benchmark (ICLR'24) that evaluates LLMs acting as agents in multiple interactive environments, with code and data in the official repo.
- AndroidWorld Mobile-OS agent benchmark — 116 real Android app tasks
Google DeepMind's AndroidWorld — 116 hand-crafted tasks across 20 real Android apps (Calendar, Files, Markor, Tasks, etc.). Evaluates LLM agents controlling a real Android device via accessibility APIs. Distinct from AndroidEnv (RL-envir…
- Atari 100k Sample-efficient RL benchmark — 100k env steps on Atari
Sample-efficiency benchmark protocol on Arcade Learning Environment (ALE) — agents limited to 100k environment steps (~2 hours of game-play) before evaluation. Introduced by SimPLe (Kaiser et al. 2019) and adopted by EfficientZero, IRIS,…
- BABILong Distributed multi-fact reasoning across long docs
Extends 20 bAbI reasoning tasks (fact chaining, induction, deduction, counting, lists) by scattering relevant sentences across PG-19 book text. Scales to 10M tokens. Tests whether models can chain distributed facts, not merely retrieve a…
- BEAM (Beyond a Million Tokens) + LIGHT Benchmark
ICLR 2026 benchmark of 100 coherent conversations up to 10M tokens with 2,000 validated probes across ten memory abilities, including contradiction resolution and event ordering. It also proposes the LIGHT episodic/working/scratchpad mem…
- Berkeley Function Calling Leaderboard (BFCL) Benchmark
Leaderboard that measures how accurately LLMs call functions (tools). It covers single-turn, multi-turn and agentic settings including web search and memory, plus hallucination and format sensitivity.
- BrowseComp Benchmark
OpenAI benchmark that measures whether browsing agents can find hard-to-locate information that may require visiting tens or hundreds of websites.
- Crafter Minecraft-inspired 2D RL benchmark with 22 achievements
Danijar Hafner's 2D Minecraft-inspired benchmark — open-world survival with 22 hierarchical achievements (collect wood → make pickaxe → mine stone → ... → collect diamond). Procedurally-generated 64x64 grid world; tests long-horizon plan…
- Evo-Memory Benchmark
Google streaming benchmark for test-time learning: agents must search, adapt and update memory across task sequences. It implements 10+ memory modules on 10 datasets and introduces the ExpRAG and ReMem baselines. Traction: Google (Ed Chi…
- EvoMemBench Benchmark
Benchmark that crosses memory scope (in-episode vs cross-episode) with content (knowledge vs execution) and compares 15 memory methods with long-context baselines under one protocol. Traction: Standardized comparison of 15 memory methods…
- GAIA Benchmark
Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset
- Habitat 3.0 (social agents) Embodied human-robot social interaction benchmark
Meta AI's Habitat 3.0 — extends Habitat 2.0 with avatar humans, social tasks (navigate-while-following, rearrangement-while-not-blocking), and human-robot collaboration evaluation. Foundational for embodied social-agent research; support…
- HaluMem Benchmark
Benchmark that measures hallucination at each stage of an agent memory system (extraction, updating, QA) rather than only final answers; HaluMem-Medium/Long datasets have ~15k memory points and contexts over 1M tokens. Traction: From Mem…
- Hanabi Learning Environment Cooperative multi-agent partial-observability card-game benchmark
DeepMind's Hanabi Learning Environment — a benchmark for cooperative multi-agent agents under partial observability and theory-of-mind. Players see each other's cards but not their own; success requires modeling teammates' beliefs. Bard …
- HELMET Holistic downstream-task long-context eval
Princeton NLP. 7 task categories — RAG with real retrieval, cited generation, summarisation, re-ranking, classification, many-shot ICL, long-document QA — at lengths up to 128K. Model-based evaluation enables open-ended scoring.
- ImplicitMemBench Implicit / procedural memory benchmark
300 items across three paradigms: procedural memory (one-shot skill retention after interference), priming (theme-driven response biases), classical conditioning (CS-US exposure shaping first decisions). Unified learning-interference-tes…
- InfiniteBench (∞Bench) 100K+ token realistic long-context comprehension
Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthet…
- LoCoMo Very long-term multi-session dialogue memory
Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), ev…
- LongBench Bilingual multitask long-context suite
Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, l…
- LongMemEval Long-term chat-assistant memory across sessions
5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to …
- MemBench Multi-dim agent memory mechanism comparison
Renmin University. Evaluates LLM-agent memory mechanisms across three axes — effectiveness (accuracy), efficiency (LLM call count), capacity (degradation as memory grows). Distinguishes factual vs reflective memory, participation vs obse…
- MemConflict Benchmark
Diagnostic benchmark covering dynamic, static and conditional memory conflicts, with black-box answer and white-box retrieval/ranking metrics; it evaluates A-Mem, LangMem, Letta, MemOS, Mem0 and Memobase. Traction: Head-to-head evaluatio…
- MemoryAgentBench 4-competency agent memory benchmark
HUST. Reconstructs existing datasets into incremental multi-turn dialogue chunks fed in time order. Plus EventQA (accurate retrieval) and FactConsolidation (selective forgetting). Targets retrieval, test-time learning, long-range underst…
- Mind2Web Benchmark
Dataset and benchmark for generalist web agents with 2,350 language-instruction tasks on 137 real websites across 31 domains.
- MineDojo Open-ended Minecraft agent benchmark + knowledge base
NVIDIA + Caltech open-ended embodied-agent benchmark on Minecraft. 3000+ programmatic + creative tasks across survival / harvest / tech-tree / combat / creative. Includes an internet-scale knowledge base (730K YouTube videos, 7K wiki pag…
- OpenAI Gym Retro Retro arcade RL benchmark — Sega / NES / SNES game integration
OpenAI's Gym Retro — wraps thousands of classic arcade games (Sega Genesis, NES, SNES, Atari 2600, Game Boy) as Gym RL environments via libretro emulators. Released 2018; powered the OpenAI Retro Contest where agents had to generalise fr…
- OSWorld Desktop OS agent benchmark — 369 tasks across Ubuntu / Windows / macOS
HKU NLP + Salesforce's OSWorld — full-desktop agent benchmark with 369 tasks spanning real applications (LibreOffice, GIMP, Chrome, VS Code, Thunderbird, Files) across Ubuntu, Windows, and macOS. Tasks require multi-application workflows…
- PROCGEN benchmark suite Procedurally-generated RL game suite — generalisation eval
OpenAI's 16-game procedurally-generated benchmark for evaluating RL generalisation. Unlike Atari (fixed levels memorisable), every PROCGEN episode uses a different procedural seed → test-set generalisation is the headline metric. Release…
- RULER Configurable multi-task long-context stress test
NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.
- SmartPlay LLM-agent benchmark across 6 games — Microsoft Research
Microsoft Research's SmartPlay — LLM-agent benchmark on 6 games (Bandits, Rock-Paper-Scissors, Tower of Hanoi, MessengerEnv, Crafter, Minecraft). Tests 9 distinct capability axes including object understanding, planning, generalisation, …
- STALE (+ CUPMEM) Benchmark
Conflict/update benchmark of 400 scenarios with 1,200 queries (up to ~150K tokens) testing whether agents notice that later facts implicitly invalidate earlier memories; the best model scores 55.2%. The CUPMEM prototype adds write-time s…
- SWE-bench Benchmark
Benchmark of real GitHub issues that coding agents must resolve with a patch, with official leaderboards for Verified (500 human-filtered instances), Lite, Multimodal and Multilingual.
- Terminal-Bench Benchmark
Benchmark of agents completing hard tasks in a terminal environment, now at version 4.0, with a public leaderboard of model and agent pairs. It is hosted by Stanford, Harbor and the Laude Institute.
- TextWorld Text-adventure RL benchmark — procedurally-generated IF games
Microsoft Research's TextWorld — a sandbox for training and evaluating RL agents on text-based games. Procedurally generates interactive-fiction games of tunable difficulty (map size, quest length, ingredients) with full ground-truth kno…
- WebArena Web-agent benchmark — 812 tasks across 6 self-hosted sites
CMU's WebArena — 812 hand-crafted web tasks across 6 self-hosted reproductions of real websites (Reddit-clone Postmill, GitLab, OneStopShop, CMS, Map, Wikipedia). Released ICLR 2024. Distinct from Mind2Web (real-website screenshots) — We…
- τ-bench (tau-bench / τ²-bench) Benchmark
Sierra's benchmark of whether agents can converse with simulated users, call tools and follow policy in domains such as retail, airline, telecom and banking. τ²-bench adds dual-control environments, and later releases add voice and knowl…
Tier 4 — early / experimental (6)
- BALROG LLM/VLM agent benchmark on 6 challenging game environments
BALROG (Benchmarking Agentic LLM and VLM Reasoning on Games) — 2024 benchmark suite covering NetHack, BabyAI, Crafter, MiniHack, TextWorld, Baba Is AI. Specifically targets long-horizon, partial-observability, exploration-heavy games whe…
- MemEvoBench Adversarial memory-evolution safety benchmark
QA tasks across 7 domains / 36 risk types plus workflow tasks adapted from 20 Agent-SafetyBench environments with noisy tool returns; both use mixed benign-and-misleading memory pools in multi-round interactions. Targets memory-evolution…
- MemoryArena Multi-session agentic memory eval gym
Stanford Digital Economy Lab et al. Benchmarks memory in realistic Memory-Agent-Environment loops where tasks span multiple sessions and subtasks are explicitly interdependent — agents must distill prior-session experiences into memory a…
- NIAH (Needle in a Haystack) Single-fact verbatim retrieval from long context
Places a known sentence ("needle") at varying positions inside a long irrelevant document ("haystack") and prompts the model to retrieve it. Tests context lengths from 2K to model max. Greg Kamradt, 2023, no formal venue.
- OSRS Bench (Old School RuneScape agent benchmark) Long-horizon MMO agent benchmark — Old School RuneScape
Community benchmark suite for LLM agents playing Old School RuneScape, a sandbox MMORPG with deep partial-observability, skill-grinding loops, and economy interaction. Tests long-horizon planning (hour-to-day quests), tool inventory mana…
- Pokemon Red benchmark (speedrun / completion) Pokemon Red completion benchmark — long-horizon JRPG agent task
Pokemon Red has emerged as a community + academic benchmark for long-horizon agentic AI. Notable instances: Peter Whiddy's PokemonRedExperiments (RL via PPO + curiosity, viral YouTube series 2023); Anthropic's ClaudePlaysPokemon Twitch s…