Catalog Systems Memory benchmarks & evaluation Mind2Web

Mind2Web

https://osu-nlp-group.github.io/Mind2Web/

Dataset and benchmark for generalist web agents with 2,350 language-instruction tasks on 137 real websites across 31 domains.

Active · Verified 2026-10-29 · fresh as of 2026-10-29 · 13 cited facts, 0 with a third-party source · 1 recorded change · Suggest a correction

At a glance

Type
Benchmark vendor fix
Tier
T3
Section
Memory benchmarks & evaluation
Created
2023-04-21 repo fix
Latest release
last commit 2025-11-05 repo fix
License
MIT (code); dataset CC-BY-4.0 vendor fix
GitHub
https://github.com/OSU-NLP-Group/Mind2Web (1036★) repo fix
Pricing
searched not found fix
Funding
searched not found fix
Freshness
2026-10-29 fix
Layer
agent-infra repo fix
Licensing
open-source vendor fix
SDK languages
python repo fix

Memory taxonomy

storage
n/a
retrieval
n/a
persistence
n/a
update
read-only
unit
n/a
governance
n/a
conflict
n/a

When to use

Optimised for: searched not found

Anti-fit: searched not found

Pros & cons

Pros

searched not found

Cons

searched not found

Claims & capabilities

Abstract: We introduce Mind2Web, the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. Existing datasets for web agents either use simulated websites or only cover a limited set of websites and tasks, thus not suitable for generalist web agents. With over 2,000 open-ended tasks collected from 137 websites spanning 31 domains and crowdsourced action sequences for the tasks, Mind2Web provides three necessary ingredients for building generalist web agents: 1) diverse domains, websites, and tasks, 2) use of real-world websites instead of simulated and simplified ones, and 3) a broad spectrum of user interaction patterns. Based on Mind2Web, we conduct an initial exploration of using large language models (LLMs) for building generalist web agents. While the raw HTML of real-world websites are often too large to be fed to LLMs, we show that first filtering it with a small LM significantly improves the effectiveness and efficiency of LLMs. Our solution demonstrates a decent level of performance, even on websites or entire domains the model has never seen before, but there is still a subs

Technical surface

API surface
searched not found
Backend storage
searched not found
Deployment
searched not found
Embedding model
searched not found
Multi-tenancy
searched not found
MCP
searched not found
A2A
searched not found
OpenTelemetry
searched not found

Similar systems

Other memory benchmarks & evaluation in the catalog, ranked by inbound references.

  • LoCoMo T3

    Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.

  • GAIA T3

    Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset

  • LongBench T3

    Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.

  • LongMemEval T3

    5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).

  • RULER T3

    NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.

  • InfiniteBench (∞Bench) T3

    Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.

Sources (12 cited facts, 0 third-party)

Each fact links to the page it came from; the badge says who published it (how to read sources).

What changed

Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)

  1. 2026-10-30 Added to the catalog #347

Related systems

Referenced by (3)