Catalog Systems Memory benchmarks & evaluation AgentBench

AgentBench

https://github.com/THUDM/AgentBench

Multi-environment benchmark (ICLR'24) that evaluates LLMs acting as agents in multiple interactive environments, with code and data in the official repo.

Active · Verified 2026-10-29 · fresh as of 2026-10-29 · 13 cited facts, 0 with a third-party source · 1 recorded change · Suggest a correction

At a glance

Type
Benchmark repo fix
Tier
T3
Section
Memory benchmarks & evaluation
Created
2023-07-28 repo fix
Latest release
last commit 2026-02-08 repo fix
License
Apache-2.0 repo fix
GitHub
https://github.com/THUDM/AgentBench (3798★) repo fix
Pricing
searched not found fix
Funding
searched not found fix
Freshness
2026-10-29 fix
Layer
agent-infra repo fix
Licensing
open-source repo fix
SDK languages
python repo fix

Memory taxonomy

storage
n/a
retrieval
n/a
persistence
n/a
update
read-only
unit
n/a
governance
n/a
conflict
n/a

When to use

Optimised for: searched not found

Anti-fit: searched not found

Pros & cons

Pros

searched not found

Cons

searched not found

Claims & capabilities

Abstract: The potential of Large Language Model (LLM) as agents has been widely acknowledged recently. Thus, there is an urgent need to quantitatively \textit{evaluate LLMs as agents} on challenging tasks in interactive environments. We present AgentBench, a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities. Our extensive test over \num API-based and open-sourced (OSS) LLMs shows that, while top commercial LLMs present a strong ability of acting as agents in complex environments, there is a significant disparity in performance between them and many OSS competitors that are no larger than 70B. We identify the typical reasons of failures in environments and LLMs, showing that poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles for developing usable LLM agents. Improving instruction following and training on high quality multi-round alignment data could improve agent performance. And different from existing assumptions, training on code present ambivalent impacts on different agent tasks. Datasets, environments, and an integrated evaluation package

Technical surface

API surface
searched not found
Backend storage
searched not found
Deployment
searched not found
Embedding model
searched not found
Multi-tenancy
searched not found
MCP
searched not found
A2A
searched not found
OpenTelemetry
searched not found

Similar systems

Other memory benchmarks & evaluation in the catalog, ranked by inbound references.

  • LoCoMo T3

    Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.

  • GAIA T3

    Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset

  • LongBench T3

    Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.

  • LongMemEval T3

    5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).

  • RULER T3

    NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.

  • InfiniteBench (∞Bench) T3

    Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.

Sources (12 cited facts, 0 third-party)

Each fact links to the page it came from; the badge says who published it (how to read sources).

What changed

Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)

  1. 2026-10-30 Added to the catalog #347

Related systems

Referenced by (1)