Memory benchmarks & evaluation
32 systems in the memory benchmarks & evaluation category of the AI Agent Infrastructure Landscape, grouped by maturity tier.
Tier 2 — production-ready (4)
- Claude Plays Pokemon (Anthropic) Anthropic agent demo — Claude 3.7/4 Sonnet plays Pokemon Red on Twitch
Anthropic's public Twitch livestream of Claude (3.7 Sonnet → Claude 4 Opus) playing Pokemon Red. Agent uses screenshot vision + Game Boy controller actions + a scratchpad memory file the model can edit. Demonstrates long-horizon agentic …
- ICLR 2026 MemAgents Workshop Workshop venue — RL + cognitive psychology + LLM memory
Full-day hybrid workshop held April 27, 2026, Rio de Janeiro. Scope: episodic and semantic memory, working memory, parametric knowledge, KGs, vector DBs, retrieval pipelines, context management, long-context utilization, temporal credit …
- MineRL Diamond Challenge Minecraft RL benchmark — obtain a diamond
MineRL is a research project / competition series using Minecraft as a long-horizon RL environment. The flagship Diamond Challenge tasks an agent with obtaining a diamond from raw world — requires deep tech-tree planning across thousands…
- NetHack Learning Environment (NLE) Roguelike RL benchmark — NetHack as agent environment
DeepMind / FAIR's NetHack Learning Environment — a Gym-compatible interface to NetHack, the canonical roguelike. Massive procedural state space, permadeath, deep item / skill interactions, ASCII partial-observability. Released at NeurIPS…
Tier 3 — emerging (22)
- AndroidWorld Mobile-OS agent benchmark — 116 real Android app tasks
Google DeepMind's AndroidWorld — 116 hand-crafted tasks across 20 real Android apps (Calendar, Files, Markor, Tasks, etc.). Evaluates LLM agents controlling a real Android device via accessibility APIs. Distinct from AndroidEnv (RL-envir…
- Atari 100k Sample-efficient RL benchmark — 100k env steps on Atari
Sample-efficiency benchmark protocol on Arcade Learning Environment (ALE) — agents limited to 100k environment steps (~2 hours of game-play) before evaluation. Introduced by SimPLe (Kaiser et al. 2019) and adopted by EfficientZero, IRIS,…
- BABILong Distributed multi-fact reasoning across long docs
Extends 20 bAbI reasoning tasks (fact chaining, induction, deduction, counting, lists) by scattering relevant sentences across PG-19 book text. Scales to 10M tokens. Tests whether models can chain distributed facts, not merely retrieve a…
- Crafter Minecraft-inspired 2D RL benchmark with 22 achievements
Danijar Hafner's 2D Minecraft-inspired benchmark — open-world survival with 22 hierarchical achievements (collect wood → make pickaxe → mine stone → ... → collect diamond). Procedurally-generated 64x64 grid world; tests long-horizon plan…
- Habitat 3.0 (social agents) Embodied human-robot social interaction benchmark
Meta AI's Habitat 3.0 — extends Habitat 2.0 with avatar humans, social tasks (navigate-while-following, rearrangement-while-not-blocking), and human-robot collaboration evaluation. Foundational for embodied social-agent research; support…
- Hanabi Learning Environment Cooperative multi-agent partial-observability card-game benchmark
DeepMind's Hanabi Learning Environment — a benchmark for cooperative multi-agent agents under partial observability and theory-of-mind. Players see each other's cards but not their own; success requires modeling teammates' beliefs. Bard …
- HELMET Holistic downstream-task long-context eval
Princeton NLP. 7 task categories — RAG with real retrieval, cited generation, summarisation, re-ranking, classification, many-shot ICL, long-document QA — at lengths up to 128K. Model-based evaluation enables open-ended scoring.
- ImplicitMemBench Implicit / procedural memory benchmark
300 items across three paradigms: procedural memory (one-shot skill retention after interference), priming (theme-driven response biases), classical conditioning (CS-US exposure shaping first decisions). Unified learning-interference-tes…
- InfiniteBench (∞Bench) 100K+ token realistic long-context comprehension
Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthet…
- LoCoMo Very long-term multi-session dialogue memory
Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), ev…
- LongBench Bilingual multitask long-context suite
Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, l…
- LongMemEval Long-term chat-assistant memory across sessions
5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to …
- MemBench Multi-dim agent memory mechanism comparison
Renmin University. Evaluates LLM-agent memory mechanisms across three axes — effectiveness (accuracy), efficiency (LLM call count), capacity (degradation as memory grows). Distinguishes factual vs reflective memory, participation vs obse…
- MemoryAgentBench 4-competency agent memory benchmark
HUST. Reconstructs existing datasets into incremental multi-turn dialogue chunks fed in time order. Plus EventQA (accurate retrieval) and FactConsolidation (selective forgetting). Targets retrieval, test-time learning, long-range underst…
- MineDojo Open-ended Minecraft agent benchmark + knowledge base
NVIDIA + Caltech open-ended embodied-agent benchmark on Minecraft. 3000+ programmatic + creative tasks across survival / harvest / tech-tree / combat / creative. Includes an internet-scale knowledge base (730K YouTube videos, 7K wiki pag…
- OpenAI Gym Retro Retro arcade RL benchmark — Sega / NES / SNES game integration
OpenAI's Gym Retro — wraps thousands of classic arcade games (Sega Genesis, NES, SNES, Atari 2600, Game Boy) as Gym RL environments via libretro emulators. Released 2018; powered the OpenAI Retro Contest where agents had to generalise fr…
- OSWorld Desktop OS agent benchmark — 369 tasks across Ubuntu / Windows / macOS
HKU NLP + Salesforce's OSWorld — full-desktop agent benchmark with 369 tasks spanning real applications (LibreOffice, GIMP, Chrome, VS Code, Thunderbird, Files) across Ubuntu, Windows, and macOS. Tasks require multi-application workflows…
- PROCGEN benchmark suite Procedurally-generated RL game suite — generalisation eval
OpenAI's 16-game procedurally-generated benchmark for evaluating RL generalisation. Unlike Atari (fixed levels memorisable), every PROCGEN episode uses a different procedural seed → test-set generalisation is the headline metric. Release…
- RULER Configurable multi-task long-context stress test
NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.
- SmartPlay LLM-agent benchmark across 6 games — Microsoft Research
Microsoft Research's SmartPlay — LLM-agent benchmark on 6 games (Bandits, Rock-Paper-Scissors, Tower of Hanoi, MessengerEnv, Crafter, Minecraft). Tests 9 distinct capability axes including object understanding, planning, generalisation, …
- TextWorld Text-adventure RL benchmark — procedurally-generated IF games
Microsoft Research's TextWorld — a sandbox for training and evaluating RL agents on text-based games. Procedurally generates interactive-fiction games of tunable difficulty (map size, quest length, ingredients) with full ground-truth kno…
- WebArena Web-agent benchmark — 812 tasks across 6 self-hosted sites
CMU's WebArena — 812 hand-crafted web tasks across 6 self-hosted reproductions of real websites (Reddit-clone Postmill, GitLab, OneStopShop, CMS, Map, Wikipedia). Released ICLR 2024. Distinct from Mind2Web (real-website screenshots) — We…
Tier 4 — early / experimental (6)
- BALROG LLM/VLM agent benchmark on 6 challenging game environments
BALROG (Benchmarking Agentic LLM and VLM Reasoning on Games) — 2024 benchmark suite covering NetHack, BabyAI, Crafter, MiniHack, TextWorld, Baba Is AI. Specifically targets long-horizon, partial-observability, exploration-heavy games whe…
- MemEvoBench Adversarial memory-evolution safety benchmark
QA tasks across 7 domains / 36 risk types plus workflow tasks adapted from 20 Agent-SafetyBench environments with noisy tool returns; both use mixed benign-and-misleading memory pools in multi-round interactions. Targets memory-evolution…
- MemoryArena Multi-session agentic memory eval gym
Stanford Digital Economy Lab et al. Benchmarks memory in realistic Memory-Agent-Environment loops where tasks span multiple sessions and subtasks are explicitly interdependent — agents must distill prior-session experiences into memory a…
- NIAH (Needle in a Haystack) Single-fact verbatim retrieval from long context
Places a known sentence ("needle") at varying positions inside a long irrelevant document ("haystack") and prompts the model to retrieve it. Tests context lengths from 2K to model max. Greg Kamradt, 2023, no formal venue.
- OSRS Bench (Old School RuneScape agent benchmark) Long-horizon MMO agent benchmark — Old School RuneScape
Community benchmark suite for LLM agents playing Old School RuneScape, a sandbox MMORPG with deep partial-observability, skill-grinding loops, and economy interaction. Tests long-horizon planning (hour-to-day quests), tool inventory mana…
- Pokemon Red benchmark (speedrun / completion) Pokemon Red completion benchmark — long-horizon JRPG agent task
Pokemon Red has emerged as a community + academic benchmark for long-horizon agentic AI. Notable instances: Peter Whiddy's PokemonRedExperiments (RL via PPO + curiosity, viral YouTube series 2023); Anthropic's ClaudePlaysPokemon Twitch s…