Atari 100k
https://arxiv.org/abs/1903.00374
Sample-efficiency benchmark protocol on Arcade Learning Environment (ALE) — agents limited to 100k environment steps (~2 hours of game-play) before evaluation. Introduced by SimPLe (Kaiser et al. 2019) and adopted by EfficientZero, IRIS, BBF, DreamerV3, Storm. Tests world-model / model-based methods where memory-as-imagination matters more than huge replay buffers.
Active · Verified 2026-10-28 · fresh as of 2026-10-28 · 29 cited facts, 0 with a third-party source · Suggest a correction
At a glance
- Tier
- T3
- Section
- Memory benchmarks & evaluation
- Created
- 2019-03 (Kaiser et al. SimPLe paper, ICLR 2020) vendor fix
- Latest release
- not applicable — not OSS fix
- License
- GPL-2.0 (ALE) vendor fix
- GitHub
- 2.3k★ C++ (ALE / atari-py) vendor fix
- Pricing
- not applicable — open benchmark; free to use vendor fix
- Funding
- not applicable — not commercial fix
- Freshness
- 2026-10-28 fix
- Layer
- agent-infra repo fix
- Licensing
- open-source vendor fix
Memory taxonomy
- storage
- n/a
- retrieval
- n/a
- persistence
- n/a
- update
- read-only
- unit
- n/a
- governance
- n/a
- conflict
- n/a
When to use
Optimised for: not applicable - eval dataset, not a system
Anti-fit: not applicable - eval dataset, not a system
Pros & cons
Pros
Tight sample budget (100k steps ≈ 2 hours) makes world-model methods competitive; long-established protocol with rich SOTA history.
Cons
Atari games are short-horizon arcade tasks — not really 'memory' benchmarks; results often overfit the 26-game subset selection.
Claims & capabilities
De-facto standard sample-efficient RL benchmark; canonical for world-model RL methods (DreamerV3 SOTA 2023; BBF surpasses human in 2023)
Technical surface
- API surface
- not applicable — eval dataset, not a system
- Backend storage
- not applicable — eval dataset, not a system
- Deployment
- not applicable — run locally
- Embedding model
- not applicable — eval dataset, not a system
- Multi-tenancy
- not applicable — eval dataset, not a system
- MCP
- not applicable — benchmark / evaluation harness
- A2A
- not applicable — benchmark / evaluation harness
- OpenTelemetry
- not applicable — benchmark / evaluation harness
Similar systems
Other memory benchmarks & evaluation in the catalog, ranked by inbound references.
- LoCoMo T3
Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.
- GAIA T3
Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset
- LongBench T3
Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.
- LongMemEval T3
5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).
- RULER T3
NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.
- InfiniteBench (∞Bench) T3
Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.
Sources (28 cited facts, 0 third-party)
Each fact links to the page it came from; the badge says who published it (how to read sources).
- What it is Sample-efficient RL benchmark — 100k env steps on Atari vendor
- What it does & what's distinct Sample-efficiency benchmark protocol on Arcade Learning Environment (ALE) — agents limit… vendor
- Claims / benefits De-facto standard sample-efficient RL benchmark; canonical for world-model RL methods (D… vendor
- Modalities visual (Atari pixel frames) — 26 game subset typical vendor
- Created 2019-03 (Kaiser et al. SimPLe paper, ICLR 2020) vendor
- License GPL-2.0 (ALE) vendor
- GitHub 2.3k★ C++ (ALE / atari-py) vendor
- Mindshare Adopted by EfficientZero, DreamerV3, IRIS, BBF, Storm, SR-SPR — canonical for sample-eff… vendor
- Citations 800+ cites — SimPLe paper (Kaiser et al.) introduced the 100k protocol vendor
- Pricing not applicable — open benchmark; free to use vendor
- Deployment not applicable — run locally vendor
- Founders / pedigree Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, et al. (Google Brain / U Warsaw / Deep… vendor
- Reproducibility Independently reproduced — multiple SOTA papers reference the protocol vendor
- Code/weights release Code public (ALE / atari-py) vendor
- Versioning not applicable — static benchmark vendor
- Tombstoning not applicable — static benchmark vendor
- Schema evolution not applicable — fixed evaluation dataset vendor
- Forgetting policy not applicable — static benchmark vendor
- Pros Tight sample budget (100k steps ≈ 2 hours) makes world-model methods competitive; long-e… vendor
- Cons Atari games are short-horizon arcade tasks — not really 'memory' benchmarks; results oft… vendor
- Project & sources arxiv 1903.00374 ALE github vendor
- cost-input-usd-per-mtok 0 vendor
- cost-output-usd-per-mtok 0 vendor
- cost-tier free vendor
- cost-pricing-model free-tier vendor
- cost-last-verified 2026-06-29 vendor
- licensing-bucket open-source vendor
- taxonomy-three-layer-rollup agent-infra repo