Catalog Systems Memory benchmarks & evaluation Crafter

Crafter

https://github.com/danijar/crafter

Danijar Hafner's 2D Minecraft-inspired benchmark — open-world survival with 22 hierarchical achievements (collect wood → make pickaxe → mine stone → ... → collect diamond). Procedurally-generated 64x64 grid world; tests long-horizon planning, exploration, credit assignment. Lightweight (single Python file) — designed as 'inner-loop Minecraft' for fast iteration. Used as flagship benchmark for DreamerV2/V3.

Research code · Verified 2026-10-28 · fresh as of 2026-10-28 · 32 cited facts, 1 with a third-party source · 1 recorded change · Suggest a correction

At a glance

Type
Minecraft-inspired 2D RL benchmark with 22 achievements repo fix
Tier
T3
Section
Memory benchmarks & evaluation
Created
2021-09 (Hafner arXiv 2109.06780) repo fix
Latest release
not applicable — not OSS fix
License
MIT repo fix
GitHub
500★ Python repo fix
Pricing
not applicable — open benchmark; free to use repo fix
Funding
not applicable — not commercial fix
Freshness
2026-10-28 fix
Layer
agent-infra repo fix
Licensing
open-source repo fix
Developer activity
OpenRank 0.96 (12-mo avg 1.01) · 3 participants — 2025-11 third-party fix
Maintainer health
1 issues in 90 days · 0 releases in 12 months · bus factor 1 — as of 2026-10-31 · concern: one contributor wrote 98% of sampled commits repo

Memory taxonomy

storage
n/a
retrieval
n/a
persistence
n/a
update
read-only
unit
n/a
governance
n/a
conflict
n/a

When to use

Optimised for: not applicable - eval dataset, not a system

Anti-fit: not applicable - eval dataset, not a system

Pros & cons

Pros

Lightweight enough to iterate fast (minutes per agent run); 22-achievement hierarchy gives interpretable progress signal; canonical for world-model RL research.

Cons

2D + small grid limits transfer to richer environments; only 22 achievements caps task ceiling; less brand recognition than 3D Minecraft benchmarks.

Claims & capabilities

Authored by DreamerV3's Hafner; canonical companion benchmark for world-model RL; lightweight enough to run on a single GPU.

Technical surface

API surface
not applicable — eval dataset, not a system
Backend storage
not applicable — eval dataset, not a system
Deployment
not applicable — run locally
Embedding model
not applicable — eval dataset, not a system
Multi-tenancy
not applicable — eval dataset, not a system
MCP
not applicable — benchmark / evaluation harness
A2A
not applicable — benchmark / evaluation harness
OpenTelemetry
not applicable — benchmark / evaluation harness

Similar systems

Other memory benchmarks & evaluation in the catalog, ranked by inbound references.

  • LoCoMo T3

    Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.

  • GAIA T3

    Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset

  • LongBench T3

    Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.

  • LongMemEval T3

    5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).

  • RULER T3

    NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.

  • InfiniteBench (∞Bench) T3

    Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.

Sources (31 cited facts, 1 third-party)

Each fact links to the page it came from; the badge says who published it (how to read sources).

What changed

Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)

  1. 2026-10-22
    • Lifecycle first recorded as “research-complete”
    #195