Evaluation & observability platforms

Production ML monitoring + LLM observability (Arize AX) — drift, eval, root-cause. Pre-LLM observability leader; now LLM-pivoted.

Arize AI's open-source LLM observability tool — OpenTelemetry-native LLM tracing, evals, datasets. Arize commercial offers Phoenix + Surveyor.

Eval-first LLM platform — datasets, eval runs, prompt playground, tracing. Founded by ex-Impira (acquired by Dropbox) team. Used by Notion, Brex, AirOps.

Galileo Evaluate / Observe / Protect — eval, tracing, guardrails; hallucination detection and ChainPoll metric. Acquired by Cisco in 2026 and now sold as Splunk Agent Observability.

Open-source DeepEval framework + Confident AI cloud — pytest-style eval runs, hallucination/relevance/bias metrics, datasets.

Open-source LLM observability and AI gateway — one-line proxy integration captures every LLM call; sessions, custom properties, evals. Acquired by Mintlify in Mar 2026 and placed in maintenance mode; helicone.ai now states it is no longe…

Collaborative LLM development platform — prompt editor, evals, tracing. UK; Index Ventures + Y Combinator backed. The team joined Anthropic in 2025 and the platform shut down on 2025-09-08.

Arize-led open OTel semantic conventions for LLM/agent tracing — auto-instrumentation for OpenAI/Anthropic/LangChain/LlamaIndex etc.

Traceloop's open-source OpenTelemetry auto-instrumentation for LLM workflows — works with any OTel-compatible backend (Datadog, New Relic, Grafana, etc.).

Enterprise eval and guardrails platform — Lynx hallucination detector (open weights), Glider judge model, Polyglot benchmark. Built by ex-Meta researchers. The evaluation platform (API, SDKs, evaluators, Lynx, GLIDER, Percival) has been …

Open-source eval framework for RAG pipelines — faithfulness, answer-relevance, context-precision, context-recall. Widely used as a baseline RAG eval kit.

ML observability with the TruLens open-source LLM eval framework — the 'RAG Triad' (context-relevance, groundedness, answer-relevance) metrics. Acquired by Snowflake 2024-05.

Open-source LLM/RAG evaluation framework — feedback functions, the RAG Triad. Originally TruEra; now Snowflake-stewarded.

Originally an enterprise LLM development platform (prompts, workflows, evals). It pivoted to a personal AI assistant (open-source vellum-assistant plus Vellum Cloud). Vellum announced its hosted Personal Assistant Platform will be shut d…

Datadog's product for evaluating, tracing and monitoring AI agents, combining offline experiments on golden datasets with production tracing and troubleshooting. Its former LLM Observability URL now redirects here.

Agent observability and evaluation platform for replaying traces step by step, debugging long-running multi-agent trajectories and evaluating live production traffic.

Open-source framework for frontier LLM evaluations from the UK AI Security Institute and Meridian Labs. It has composable datasets, agents, tools and scorers, 200+ prebuilt evals including agentic tasks, and a log viewer.

Open-source agent observability platform that captures LLM calls, tool calls, sub-agents, costs and tokens. Its Signals feature clusters agent failures automatically, and it also runs evals.

OpenAI's open-source framework and registry of benchmarks for evaluating LLMs and LLM systems.

Prompt versioning + tracing platform — non-eng-friendly prompt editor; early entrant pre-LLM-ops wave.

OpenTelemetry-based AI observability platform that traces model calls, agent steps, API requests and database queries in one view. SDKs are available for Python, JS/TS and Rust.