Inspect AI
Open-source framework for frontier LLM evaluations from the UK AI Security Institute and Meridian Labs. It has composable datasets, agents, tools and scorers, 200+ prebuilt evals including agentic tasks, and a log viewer.
Active · Verified 2026-10-29 · fresh as of 2026-10-29 · 15 cited facts, 0 with a third-party source · 1 recorded change · Suggest a correction
At a glance
- Tier
- T3
- Created
- 2023-11-14 repo fix
- Latest release
- last commit 2026-10-29 repo fix
- Pricing
- searched not found fix
- Funding
- searched not found fix
- Freshness
- 2026-10-29 fix
- Layer
- agent-infra repo fix
- Licensing
- open-source vendor fix
- Protocols
- mcp-client vendor fix
When to use
Optimised for: searched not found
Anti-fit: searched not found
Pros & cons
Pros
searched not found
Cons
searched not found
Claims & capabilities
An open-source framework for large language model evaluations
Technical surface
- API surface
- searched not found
- Backend storage
- searched not found
- Deployment
- searched not found
- Embedding model
- searched not found
- Multi-tenancy
- searched not found
- MCP
- searched not found
- A2A
- searched not found
- OpenTelemetry
- searched not found
Similar systems
Other evaluation & observability platforms in the catalog, ranked by inbound references.
- Arize Phoenix T1
Arize AI's open-source LLM observability tool — OpenTelemetry-native LLM tracing, evals, datasets. Arize commercial offers Phoenix + Surveyor.
- Arize AI T1
Production ML monitoring + LLM observability (Arize AX) — drift, eval, root-cause. Pre-LLM observability leader; now LLM-pivoted.
- Braintrust T1
Eval-first LLM platform — datasets, eval runs, prompt playground, tracing. Founded by ex-Impira (acquired by Dropbox) team. Used by Notion, Brex, AirOps.
- Confident AI / DeepEval T2
Open-source DeepEval framework + Confident AI cloud — pytest-style eval runs, hallucination/relevance/bias metrics, datasets.
- OpenLLMetry (Traceloop) T2
Traceloop's open-source OpenTelemetry auto-instrumentation for LLM workflows — works with any OTel-compatible backend (Datadog, New Relic, Grafana, etc.).
- Helicone T2
Open-source LLM observability and AI gateway — one-line proxy integration captures every LLM call; sessions, custom properties, evals. Acquired by Mintlify in Mar 2026 and placed in maintenance mode; helicone.ai now states it is no longer part of Mintlify.
Sources (14 cited facts, 0 third-party)
Each fact links to the page it came from; the badge says who published it (how to read sources).
- What it is Framework vendor
- What it does & what's distinct Open-source framework for frontier LLM evaluations from the UK AI Security Institute and… vendor
- Claims / benefits An open-source framework for large language model evaluations vendor
- Created 2023-11-14 repo
- Latest release last commit 2026-10-29 repo
- License MIT vendor
- GitHub https://github.com/UKGovernmentBEIS/inspect_ai (3128★) repo
- Mindshare 3128 GitHub stars repo
- Project & sources https://inspect.aisi.org.uk/ ; https://github.com/UKGovernmentBEIS/inspect_ai vendor
- licensing-bucket open-source vendor
- taxonomy-three-layer-rollup agent-infra repo
- sdk-languages python registry
- protocol-support mcp-client vendor
- maturity beta registry
What changed
Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)
- 2026-10-30 Added to the catalog #347
Related systems
References (18)
- Anthropic Claude (foundation models) integrates with — Inspect AI model-providers page documents a Anthropic provider. (verified 2026-10-29)
- AWS SageMaker integrates with — Inspect AI model-providers page documents a AWS SageMaker provider. (verified 2026-10-29)
- Claude Code (Anthropic) integrates with — Inspect AI "Agent Bridge" page documents bridging Claude Code agents into Inspect evaluations. (verified 2026-10-29)
- DeepSeek R1 / V3 family integrates with — Inspect AI model-providers page documents a DeepSeek provider. (verified 2026-10-29)
- Fireworks AI integrates with — Inspect AI model-providers page documents a Fireworks AI provider. (verified 2026-10-29)
- Google Gemini 3 family integrates with — Inspect AI model-providers page documents a Google Gemini provider. (verified 2026-10-29)
- LangChain (framework) integrates with — Inspect AI "Agent Bridge" page documents bridging LangChain agents into Inspect evaluations. (verified 2026-10-29)
- LiteLLM integrates with — Inspect AI model-providers page documents a LiteLLM Proxy provider. (verified 2026-10-29)
- Mistral Large 2 / Mixtral family integrates with — Inspect AI model-providers page documents a Mistral provider. (verified 2026-10-29)
- OpenAI Agents SDK integrates with — Inspect AI "Agent Bridge" page documents bridging OpenAI Agents SDK agents into Inspect evaluations. (verified 2026-10-29)
- OpenAI Codex CLI integrates with — Inspect AI "Agent Bridge" page documents bridging Codex CLI agents into Inspect evaluations. (verified 2026-10-29)
- OpenAI GPT family (GPT-5 / GPT-4o / o3 / o4) integrates with — Inspect AI model-providers page documents a OpenAI provider. (verified 2026-10-29)
- OpenRouter integrates with — Inspect AI model-providers page documents a OpenRouter provider. (verified 2026-10-29)
- Pydantic-AI integrates with — Inspect AI "Agent Bridge" page documents bridging Pydantic AI agents into Inspect evaluations. (verified 2026-10-29)
- SGLang integrates with — Inspect AI model-providers page documents a SGLang provider. (verified 2026-10-29)
- Together AI integrates with — Inspect AI model-providers page documents a Together AI provider. (verified 2026-10-29)
- vLLM integrates with — Inspect AI model-providers page documents a vLLM provider. (verified 2026-10-29)
- xAI Grok 4 integrates with — Inspect AI model-providers page documents a Grok (xAI) provider. (verified 2026-10-29)