SWE-bench
Benchmark of real GitHub issues that coding agents must resolve with a patch, with official leaderboards for Verified (500 human-filtered instances), Lite, Multimodal and Multilingual.
Active · Verified 2026-10-29 · fresh as of 2026-10-29 · 14 cited facts, 0 with a third-party source · 1 recorded change · Suggest a correction
At a glance
- Tier
- T3
- Section
- Memory benchmarks & evaluation
- Created
- 2023-10-04 repo fix
- Latest release
- last commit 2026-09-02 repo fix
- GitHub
- https://github.com/SWE-bench/SWE-bench (6117★) repo fix
- Pricing
- searched not found fix
- Funding
- searched not found fix
- Freshness
- 2026-10-29 fix
- Layer
- agent-infra repo fix
- Licensing
- open-source vendor fix
Memory taxonomy
- storage
- n/a
- retrieval
- n/a
- persistence
- n/a
- update
- read-only
- unit
- n/a
- governance
- n/a
- conflict
- n/a
When to use
Optimised for: searched not found
Anti-fit: searched not found
Pros & cons
Pros
searched not found
Cons
searched not found
Claims & capabilities
Abstract: Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is abl
Technical surface
- API surface
- searched not found
- Backend storage
- searched not found
- Deployment
- searched not found
- Embedding model
- searched not found
- Multi-tenancy
- searched not found
- MCP
- searched not found
- A2A
- searched not found
- OpenTelemetry
- searched not found
Similar systems
Other memory benchmarks & evaluation in the catalog, ranked by inbound references.
- LoCoMo T3
Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.
- GAIA T3
Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset
- LongBench T3
Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.
- LongMemEval T3
5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).
- RULER T3
NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.
- InfiniteBench (∞Bench) T3
Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.
Sources (13 cited facts, 0 third-party)
Each fact links to the page it came from; the badge says who published it (how to read sources).
- What it is Benchmark vendor
- What it does & what's distinct Benchmark of real GitHub issues that coding agents must resolve with a patch, with offic… vendor
- Claims / benefits Abstract: Language models have outpaced our ability to evaluate them effectively, but fo… paper
- Created 2023-10-04 repo
- Latest release last commit 2026-09-02 repo
- License MIT vendor
- GitHub https://github.com/SWE-bench/SWE-bench (6117★) repo
- Mindshare 6117 GitHub stars repo
- Project & sources https://www.swebench.com/ ; https://github.com/SWE-bench/SWE-bench ; https://arxiv.org/a… vendor
- eval-weights-and-biases-agent yes vendor
- licensing-bucket open-source vendor
- taxonomy-three-layer-rollup agent-infra repo
- sdk-languages python registry
What changed
Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)
- 2026-10-30 Added to the catalog #347