| System | Status | Tier | Section | What it is | OTel | Deployment | Pricing | License | Latest release | GitHub | Verified |
|---|---|---|---|---|---|---|---|---|---|---|---|
Active | T1 | Evaluation & observability platforms | ML + LLM observability platform ↗ 2 | Both (self-hosted + managed cloud) — AX SaaS; self-hosted on AX Enterprise only ↗ 2 | AX Free $0/mo (25k spans/mo); AX Pro $50/mo (50k spans/mo); AX Enterprise custom (SaaS or self-hosted); Phoenix OSS free ↗ 2 | proprietary; Python SDK Apache-2.0 ↗ ● | v8.61.0 (2026-10-26) — arize ↗ 2 | 2026-10-27 | |||
Active | T1 | Evaluation & observability platforms | OSS LLM tracing + eval ↗ 2 | Elastic-2.0 ↗ 2 | v20.26.1 (2026-10-25) ↗ 2 | Arize-ai/phoenix — ~12.0k stars, last commit 2026-10 ↗ ● | 2026-10-27 | ||||
Active | T1 | Evaluation & observability platforms | Eval + observability for AI ↗ 2 | Both (self-hosted + managed cloud) — hosted SaaS; on-prem deployment on Enterprise ↗ 2 | Starter $0/mo; Pro $249/mo; Enterprise custom ↗ 2 | proprietary; SDKs open source ↗ 2 | v0.48.1 (2026-10-26) — braintrust Python SDK ↗ 2 | 2026-10-27 | |||
Acquired | T1 | Evaluation & observability platforms | Eval + observability + guardrails ↗ 2 | Both (self-hosted + managed cloud) — SaaS, VPC or on-premises ↗ 2 | Splunk Agent Observability from $100/mo per 1.2M spans billed annually (includes 500M Luna tokens); free edition available ↗ 2 | proprietary; Python SDK Apache-2.0 ↗ ● | v2.7.0 (2026-10-14) — galileo ↗ 2 | rungalileo/galileo-python — ~22 stars, last commit 2026-10 ↗ ● | 2026-10-28 | ||
Acquired | T1 | Memory observability & monitoring | Real-time intent / belief drift detection ↗ 2 | first-class — OTel-native ingestion ↗ 2 | SaaS cloud; Enterprise: VPC or on-prem ↗ 2 | Standalone pricing retired — galileo.ai/pricing now redirects to Splunk pricing; sold as Splunk Agent Observability ↗ 2 | · | · | · | 2026-10-28 | |
Acquired | T1 | Memory observability & monitoring | Memory ops as named spans + Agent Graphs ↗ 2 | first-class — native OTel SDK ↗ 2 | SaaS cloud (Berlin + SF) or self-hosted Docker/K8s; ClickHouse native integration post-acquisition ↗ 2 | free: 50K observations/mo; Pro: $59/mo; Team: $199/mo; Enterprise: custom ↗ 2 | MIT (core; ee/ directories under separate enterprise license) ↗ ● | v4.27.0 (Sep 1; page shows no year, presumably 2026) ↗ ● | langfuse/langfuse — ~36.1k stars, last release 2026-09 ↗ ● | 2026-10-29 | |
Active | T1 | Memory observability & monitoring | Memory mutations as distinct span types ↗ 2 | first-class — OTel ingestion + export ↗ 2 | SaaS cloud (US); Enterprise: dedicated tenant ↗ 2 | Developer: free (5K traces/mo); Plus: $39/seat/mo (10K traces); Enterprise: custom (SSO/SAML; dedicated support) ↗ 2 | · | SDK v0.14.11 (2026-10-16) — platform itself proprietary ↗ 2 | · | 2026-10-23 | |
Active | T2 | Memory observability & monitoring | Operational metrics for agent memory ↗ 2 | first-class — OTel-native ↗ 2 | SaaS cloud; SDK integration into agent code ↗ 2 | free tier; Pro/Enterprise pricing not publicly listed; contact sales ↗ 2 | MIT (SDK) ↗ 2 | v0.4.21 (2025-08-29) ↗ 2 | AgentOps-AI/agentops — ~5.8k stars ↗ ● | 2026-10-22 | |
Active | T2 | Memory benchmarks & evaluation | Anthropic agent demo — Claude 3.7/4 Sonnet plays Pokemon Red on Twitch ↗ 2 | · | not applicable — research demo ↗ 2 | not applicable — research demo / community project ↗ 2 | · | Twitch livestream ongoing; runs distinct per Claude model version ↗ 2 | · | 2026-10-28 | |
Active | T2 | Evaluation & observability platforms | LLM eval framework + cloud ↗ 2 | Both (self-hosted + managed cloud) — Confident AI cloud; dedicated on-prem on Enterprise; DeepEval runs locally ↗ 2 | Free $0; Starter $200/mo; Team $2,000/mo; Enterprise custom ↗ 2 | Apache-2.0 (DeepEval) ↗ 2 | v4.3.1 (2026-10-16) — deepeval ↗ 2 | confident-ai/deepeval — ~19.1k stars, last commit 2026-10 ↗ ● | 2026-10-27 | ||
Maintenance | T2 | Evaluation & observability platforms | LLM observability + gateway ↗ 2 | Both (self-hosted + managed cloud) — Docker Compose self-host; Helm and on-prem for Enterprise ↗ ● | Hobby free (10k requests); Pro $79/mo; Team $799/mo; Enterprise custom; plus usage-based pricing ↗ 2 | Apache-2.0 (Core) ↗ 2 | v1.8.3 (2025-11-07) — @helicone/helpers ↗ 2 | Helicone/helicone — ~6.2k stars, last commit 2026-10 ↗ ● | 2026-10-29 | ||
Archived | T2 | Evaluation & observability platforms | LLM eval + collab ↗ 2 | v0.8.38 (2025-05-09) — humanloop SDK, final ↗ 2 | humanloop/humanloop-python — ~12 stars, last commit 2025-05 ↗ ● | 2026-10-29 | |||||
Active | T2 | Memory benchmarks & evaluation | Workshop venue — RL + cognitive psychology + LLM memory ↗ 2 | · | · | · | · | · | · | 2026-10-23 | |
Active | T2 | Memory benchmarks & evaluation | Minecraft RL benchmark — obtain a diamond ↗ 2 | · | not applicable — run locally ↗ 2 | not applicable — open benchmark; free to use ↗ 2 | MIT ↗ 2 | · | ~700★ Python ↗ 2 | 2026-05-14 | |
Active | T2 | Memory benchmarks & evaluation | Roguelike RL benchmark — NetHack as agent environment ↗ ● | · | not applicable — run locally ↗ ● | not applicable — open benchmark; free to use ↗ ● | NetHack General Public License ↗ ● | v0.9.x series (active as of 2024) ↗ ● | 1.0k★ Python/C ↗ ● | 2026-05-14 | |
Active | T2 | Evaluation & observability platforms | OTel semantic conventions for LLMs ↗ ● | Self-hosted only — library ↗ ● | Apache-2.0 ↗ ● | v0.1.42 (2026-10-23) — openinference-semantic-conventions ↗ 2 | Arize-ai/openinference — ~1.3k stars, last commit 2026-10 ↗ ● | 2026-10-27 | |||
Active | T2 | Evaluation & observability platforms | OTel auto-instrumentation for LLMs ↗ ● | Self-hosted only — library ↗ ● | Apache-2.0 ↗ ● | v0.62.4 (2026-09-29) — traceloop-sdk ↗ 2 | traceloop/openllmetry — ~7.5k stars, last commit 2026-10 ↗ ● | 2026-10-27 | |||
Archived | T2 | Evaluation & observability platforms | Enterprise eval + guardrails ↗ 2 | Contact sales — no public price list ↗ 2 | proprietary; Python SDK MIT ↗ 2 | v0.1.26 (2026-09-17) — patronus ↗ 2 | 2026-10-29 | ||||
Active | T2 | Evaluation & observability platforms | RAG eval metrics ↗ ● | Self-hosted only — library ↗ ● | Apache-2.0 ↗ ● | v0.4.3 (2026-01-13) — ragas ↗ ● | vibrantlabsai/ragas — ~16.2k stars, last commit 2026-02 ↗ ● | 2026-10-28 | |||
Active | T2 | Memory observability & monitoring | Offline scanner — memory poisoning detector ↗ ● | local CLI — Python; pip install; no server required ↗ ● | free / open source (MIT); no paid tier ↗ ● | MIT ↗ ● | v0.1.0 2026-04-02 ↗ ● | 1★ Python ↗ ● | 2026-10-22 | ||
Acquired | T2 | Evaluation & observability platforms | ML observability + RAG triad ↗ 2 | v12.6.1 (2024-03-04) — truera (Python SDK) ↗ 2 | 2026-10-29 | ||||||
Active | T2 | Evaluation & observability platforms | LLM eval framework ↗ ● | Self-hosted only — library ↗ 2 | MIT ↗ ● | v2.16.0 (2026-10-19) — trulens ↗ 2 | truera/trulens — ~3.6k stars, last commit 2026-10 ↗ ● | 2026-10-29 | |||
Shut down | T2 | Evaluation & observability platforms | LLM dev platform (build + eval + deploy) ↗ 2 | proprietary; Python SDK MIT ↗ 2 | v1.14.8 (2026-10-02) — vellum-ai (Python SDK) ↗ 2 | 2026-10-29 | |||||
Active | T3 | Memory benchmarks & evaluation | Benchmark ↗ ● | Apache-2.0 ↗ ● | last commit 2026-02-08 ↗ ● | https://github.com/THUDM/AgentBench (3798★) ↗ ● | 2026-10-29 | ||||
Active | T3 | Memory benchmarks & evaluation | Mobile-OS agent benchmark — 116 real Android app tasks ↗ ● | · | not applicable — run locally ↗ ● | not applicable — open benchmark; free to use ↗ ● | Apache-2.0 ↗ ● | · | 1.2k★ Python ↗ ● | 2026-10-28 | |
Active | T3 | Memory benchmarks & evaluation | Sample-efficient RL benchmark — 100k env steps on Atari ↗ 2 | · | not applicable — run locally ↗ 2 | not applicable — open benchmark; free to use ↗ 2 | GPL-2.0 (ALE) ↗ 2 | · | 2.3k★ C++ (ALE / atari-py) ↗ 2 | 2026-10-28 | |
Active | T3 | Memory benchmarks & evaluation | Distributed multi-fact reasoning across long docs ↗ 2 | · | not applicable — run locally ↗ ● | not applicable — open benchmark; free to use ↗ ● | Apache-2.0 ↗ ● | 248★ Jupyter Notebook ↗ ● | 2026-10-28 | ||
Active | T3 | Memory benchmarks & evaluation | Benchmark ↗ 2 | MIT code; CC BY-SA 4.0 data ↗ 2 | last commit 2026-08-31 ↗ ● | https://github.com/mohammadtavakoli78/BEAM (164★) ↗ ● | 2026-10-23 | ||||
Active | T3 | Memory benchmarks & evaluation | Benchmark ↗ 2 | Apache-2.0 ↗ 2 | last commit 2026-03-23 ↗ ● | https://github.com/ShishirPatil/gorilla (13063★) ↗ ● | 2026-10-29 | ||||
Active | T3 | Memory benchmarks & evaluation | Benchmark ↗ 2 | MIT ↗ 2 | last commit 2026-04-22 ↗ ● | https://github.com/openai/simple-evals (4686★) ↗ ● | 2026-10-29 | ||||
Research code | T3 | Memory benchmarks & evaluation | Minecraft-inspired 2D RL benchmark with 22 achievements ↗ ● | · | not applicable — run locally ↗ ● | not applicable — open benchmark; free to use ↗ ● | MIT ↗ ● | · | 500★ Python ↗ ● | 2026-10-28 | |
Active | T3 | Evaluation & observability platforms | Product ↗ 2 | yes ↗ 2 | proprietary ↗ 2 | · | 2026-10-29 |