Evaluation & observability platforms
15 systems in the evaluation & observability platforms category of the AI Agent Infrastructure Landscape, grouped by maturity tier.
Tier 1 — battle-tested (4)
- Arize AI ML + LLM observability platform
Production ML monitoring + LLM observability (Arize AX) — drift, eval, root-cause. Pre-LLM observability leader; now LLM-pivoted.
- Arize Phoenix OSS LLM tracing + eval
Arize AI's open-source LLM observability tool — OpenTelemetry-native LLM tracing, evals, datasets. Arize commercial offers Phoenix + Surveyor.
- Braintrust Eval + observability for AI
Eval-first LLM platform — datasets, eval runs, prompt playground, tracing. Founded by ex-Impira (acquired by Dropbox) team. Used by Notion, Brex, AirOps.
- Galileo Eval + observability + guardrails
Galileo Evaluate / Observe / Protect — eval, tracing, guardrails. Big focus on hallucination detection and ChainPoll evaluator metric.
Tier 2 — production-ready (10)
- Confident AI / DeepEval LLM eval framework + cloud
Open-source DeepEval framework + Confident AI cloud — pytest-style eval runs, hallucination/relevance/bias metrics, datasets.
- Helicone LLM observability + gateway
Open-source LLM observability with one-line proxy integration — gateway routes through their proxy, captures every LLM call. Sessions + custom properties + evals.
- Humanloop LLM eval + collab
Collaborative LLM development platform — non-eng prompt editor, eval, tracing. UK; Index Ventures + Y Combinator backed.
- OpenInference OTel semantic conventions for LLMs
Arize-led open OTel semantic conventions for LLM/agent tracing — auto-instrumentation for OpenAI/Anthropic/LangChain/LlamaIndex etc.
- OpenLLMetry (Traceloop) OTel auto-instrumentation for LLMs
Traceloop's open-source OpenTelemetry auto-instrumentation for LLM workflows — works with any OTel-compatible backend (Datadog, New Relic, Grafana, etc.).
- Patronus AI Enterprise eval + guardrails
Enterprise eval and guardrails platform — Lynx hallucination detector (open weights), Glider judge model, Polyglot benchmark. Built by ex-Meta researchers.
- Ragas RAG eval metrics
Open-source eval framework for RAG pipelines — faithfulness, answer-relevance, context-precision, context-recall. Widely used as a baseline RAG eval kit.
- TruEra ML observability + RAG triad
ML observability with the TruLens open-source LLM eval framework — the 'RAG Triad' (context-relevance, groundedness, answer-relevance) metrics. Acquired by Snowflake 2024-05.
- TruLens LLM eval framework
Open-source LLM/RAG evaluation framework — feedback functions, the RAG Triad. Originally TruEra; now Snowflake-stewarded.
- Vellum AI LLM dev platform (build + eval + deploy)
End-to-end LLM development platform — eval, prompt management, deployment, observability. Targeted at enterprise teams shipping LLM products.
Tier 3 — emerging (1)
- PromptLayer Prompt management + tracing
Prompt versioning + tracing platform — non-eng-friendly prompt editor; early entrant pre-LLM-ops wave.