OpenAI Evals

OpenAI's open-source framework and registry of benchmarks for evaluating LLMs and LLM systems.

Active · Verified 2026-10-29 · fresh as of 2026-10-29 · 14 cited facts, 0 with a third-party source · 1 recorded change · Suggest a correction

At a glance

Optimised for: searched not found

Anti-fit: searched not found

searched not found

searched not found

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Other evaluation & observability platforms in the catalog, ranked by inbound references.

Arize Phoenix T1
Arize AI's open-source LLM observability tool — OpenTelemetry-native LLM tracing, evals, datasets. Arize commercial offers Phoenix + Surveyor.
Arize AI T1
Production ML monitoring + LLM observability (Arize AX) — drift, eval, root-cause. Pre-LLM observability leader; now LLM-pivoted.
Braintrust T1
Eval-first LLM platform — datasets, eval runs, prompt playground, tracing. Founded by ex-Impira (acquired by Dropbox) team. Used by Notion, Brex, AirOps.
Confident AI / DeepEval T2
Open-source DeepEval framework + Confident AI cloud — pytest-style eval runs, hallucination/relevance/bias metrics, datasets.
OpenLLMetry (Traceloop) T2
Traceloop's open-source OpenTelemetry auto-instrumentation for LLM workflows — works with any OTel-compatible backend (Datadog, New Relic, Grafana, etc.).
Helicone T2
Open-source LLM observability and AI gateway — one-line proxy integration captures every LLM call; sessions, custom properties, evals. Acquired by Mintlify in Mar 2026 and placed in maintenance mode; helicone.ai now states it is no longer part of Mintlify.

Each fact links to the page it came from; the badge says who published it (how to read sources).

Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)

Weights & Biases integrates with — OpenAI Evals README says evals can also be run and created using Weights & Biases. (verified 2026-10-29)