Catalog Systems Memory benchmarks & evaluation Pokemon Red benchmark (speedrun / completion)

Pokemon Red benchmark (speedrun / completion)

https://github.com/PWhiddy/PokemonRedExperiments

Pokemon Red has emerged as a community + academic benchmark for long-horizon agentic AI. Notable instances: Peter Whiddy's PokemonRedExperiments (RL via PPO + curiosity, viral YouTube series 2023); Anthropic's ClaudePlaysPokemon Twitch stream (2025-02); Google's GeminiPlaysPokemon (2025); MorphLabs PokemonRedRL. The benchmark protocol varies: badge count, gym progression, full Elite-Four completion. Common across all: ~5k-20k+ step horizon, sparse rewards, deep partial-observability.

Active · Verified 2026-10-28 · fresh as of 2026-10-28 · 31 cited facts, 1 with a third-party source · Suggest a correction

At a glance

Type
Pokemon Red completion benchmark — long-horizon JRPG agent task repo fix
Tier
T4
Section
Memory benchmarks & evaluation
Created
2023-09 (Whiddy YouTube + GitHub release) repo fix
Latest release
not applicable — not OSS fix
License
MIT (Whiddy repo) repo fix
GitHub
7.6k★ Python (PokemonRedExperiments) repo fix
Pricing
not applicable — open benchmark; free to use repo fix
Funding
not applicable — not commercial fix
Freshness
2026-10-28 fix
Layer
agent-infra repo fix
Licensing
open-source repo fix
Developer activity
OpenRank 0.14 (12-mo avg 1.98) · 1 participants — 2026-09 third-party fix

Memory taxonomy

storage
n/a
retrieval
n/a
persistence
n/a
update
read-only
unit
n/a
governance
n/a
conflict
n/a

When to use

Optimised for: not applicable - eval dataset, not a system

Anti-fit: not applicable - eval dataset, not a system

Pros & cons

Pros

Most-watched public AI-agent benchmark via Twitch streams; ~hours-to-days horizon stresses long-horizon planning; deeply community-engaging; multiple frontier labs racing on it.

Cons

No standardized scoring protocol (badge-count vs completion vs speed); Game Boy ROM legality issues; Twitch streams not always reproducible at the playthrough level.

Claims & capabilities

Anthropic ClaudePlaysPokemon: 2/8 badges over ~80hr (Claude 3.7 Sonnet, Feb 2025); Google Gemini 2.5 Pro reportedly completed full game; Whiddy's PPO+curiosity RL completed up to Pewter Gym; viral mindshare via Twitch livestreams

Technical surface

API surface
not applicable — eval dataset, not a system
Backend storage
not applicable — eval dataset, not a system
Deployment
not applicable — run locally
Embedding model
not applicable — eval dataset, not a system
Multi-tenancy
not applicable — eval dataset, not a system
MCP
not applicable — benchmark / evaluation harness
A2A
not applicable — benchmark / evaluation harness
OpenTelemetry
not applicable — benchmark / evaluation harness

Similar systems

Other memory benchmarks & evaluation in the catalog, ranked by inbound references.

  • LoCoMo T3

    Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.

  • GAIA T3

    Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset

  • LongBench T3

    Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.

  • LongMemEval T3

    5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).

  • RULER T3

    NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.

  • InfiniteBench (∞Bench) T3

    Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.

Sources (30 cited facts, 1 third-party)

Each fact links to the page it came from; the badge says who published it (how to read sources).