Pokemon Red benchmark (speedrun / completion)
https://github.com/PWhiddy/PokemonRedExperiments
Pokemon Red has emerged as a community + academic benchmark for long-horizon agentic AI. Notable instances: Peter Whiddy's PokemonRedExperiments (RL via PPO + curiosity, viral YouTube series 2023); Anthropic's ClaudePlaysPokemon Twitch stream (2025-02); Google's GeminiPlaysPokemon (2025); MorphLabs PokemonRedRL. The benchmark protocol varies: badge count, gym progression, full Elite-Four completion. Common across all: ~5k-20k+ step horizon, sparse rewards, deep partial-observability.
At a glance
- Type
- Pokemon Red completion benchmark — long-horizon JRPG agent task
- Tier
- T4
- Section
- Memory benchmarks & evaluation
- Created
- 2023-09 (Whiddy YouTube + GitHub release)
- Latest release
- not applicable — not OSS
- License
- MIT (Whiddy repo)
- Pricing
- not applicable — open benchmark; free to use
- Funding
- not applicable — not commercial
Taxonomy
- storage
- n/a
- retrieval
- n/a
- persistence
- n/a
- update
- read-only
- unit
- n/a
- governance
- n/a
- conflict
- n/a
When to use
Optimised for: not applicable - eval dataset, not a system
Anti-fit: not applicable - eval dataset, not a system
Pros & cons
Pros
Most-watched public AI-agent benchmark via Twitch streams; ~hours-to-days horizon stresses long-horizon planning; deeply community-engaging; multiple frontier labs racing on it.
Cons
No standardized scoring protocol (badge-count vs completion vs speed); Game Boy ROM legality issues; Twitch streams not always reproducible at the playthrough level.
Claims & capabilities
Anthropic ClaudePlaysPokemon: 2/8 badges over ~80hr (Claude 3.7 Sonnet, Feb 2025); Google Gemini 2.5 Pro reportedly completed full game; Whiddy's PPO+curiosity RL completed up to Pewter Gym; viral mindshare via Twitch livestreams
Technical surface
- API surface
- not applicable — eval dataset, not a system
- Backend storage
- not applicable — eval dataset, not a system
- Deployment
- not applicable — run locally
- Embedding model
- not applicable — eval dataset, not a system
- Multi-tenancy
- not applicable — eval dataset, not a system
- MCP
- not applicable — benchmark / evaluation harness
- A2A
- not applicable — benchmark / evaluation harness
- OpenTelemetry
- not applicable — benchmark / evaluation harness
Similar systems
Other memory benchmarks & evaluation in the catalog, ranked by inbound references.
- LoCoMo T3
Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.
- LongBench T3
Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.
- LongMemEval T3
5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).
- RULER T3
NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.
- InfiniteBench (∞Bench) T3
Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.
- Atari 100k T3
Sample-efficiency benchmark protocol on Arcade Learning Environment (ALE) — agents limited to 100k environment steps (~2 hours of game-play) before evaluation. Introduced by SimPLe (Kaiser et al. 2019) and adopted by EfficientZero, IRIS, BBF, DreamerV3, Storm. Tests world-model / model-based methods where memory-as-imagination matters more than huge replay buffers.