Observability adoption
82.6%
of 132 analyzed catalog products ship at least one observability integration. Industry headline: 89% (LangChain 2025).
Evaluation gap (the next frontier after observability)
LangChain's State of Agent Engineering 2025 found that 89% of practitioners have observability adopted but only 52% have evals — a 37-point structural gap. The Berkeley RDI MAP study (Q1 2026) corroborates: 74% rely primarily on human evaluation. The Datadog State of AI Agents 2026 report lists "reliable evaluation loops" as a top recommendation. This view answers the catalog-level question "how do I know if my agent is actually getting better?" — the question that had no catalog-level surface before T1-2.
This view's distinctive value is the gap. Every comparable catalog tracks eval tooling on its own. We're the first to surface "products that have observability but no eval" row-by-row — the structural failure mode the LangChain survey identified.
Out of scope: eval methodology comparison (LLM-as-judge vs human rubric vs canary set) and benchmark scores (T1-4 territory). This view tracks integration support only — which products plug into which eval tooling.
Eval adoption
44.7%
of 132 analyzed catalog products ship at least one eval integration. Industry headline: 52% (LangChain 2025).
Eval orphans
42
products with observability ≥1 tool but zero eval tools. These are the structural failures the LangChain survey identified. Use the "only eval-orphans" filter below to see them.
Legend
- yes eval tool integration documented — citation in cell tooltip
- no eval tool absent / not exposed for this product class
- unknown not yet researched, depth-floor reached, or not applicable
- orphan product has observability ≥1 but zero eval tools
Matrix — showing 132 of 1002
| Product | Tier | Obs | Eval | Gap | Orphan | LangSmith Evals | Braintrust | W&B Agent Eval | Helicone Evals | Custom test harness | Human-in-loop | Prod traffic replay | Section |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Google Gemini Memory | T1 | 8 | 0 | +8 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Platform-provider memory |
| Vertex AI Memory Bank (now Agent Platform Memory Bank) Custom test harness | T2 | 8 | 1 | +7 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Platform-provider memory | |
| Mistral Le Chat — Memories (now Vibe) | T1 | 7 | 0 | +7 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Platform-provider memory |
| Mistral Vibe (Remote Agents) + Mistral Medium 3.5 | T1 | 7 | 0 | +7 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Coding-agent memory |
| Anthropic Claude Memory | T1 | 6 | 0 | +6 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Platform-provider memory |
| Anthropic Claude (foundation models) LangSmith Evals | T1 | 8 | 4 | +4 | ● | ● | ● | ● | ○ | ○ | ○ | Foundation models (substrate reference) | |
| OpenAI Realtime API LangSmith Evals | T1 | 8 | 4 | +4 | ● | ● | ● | ● | ○ | ○ | ○ | Voice agent platforms | |
| Pydantic Logfire Braintrust | T3 | 5 | 1 | +4 | – | ● | – | – | – | – | – | Evaluation & observability platforms | |
| Grok Memory (xAI) | T1 | 4 | 0 | +4 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Platform-provider memory |
| Anthropic Computer Use LangSmith Evals | T1 | 8 | 5 | +3 | ● | ● | ● | ● | ● | ○ | ○ | Computer-use & desktop agents | |
| OpenAI Agents SDK LangSmith Evals | T1 | 8 | 5 | +3 | ● | ● | ● | ● | ● | ○ | ○ | Agent frameworks (no first-party memory layer) | |
| agentgateway Braintrust | T3 | 5 | 2 | +3 | – | ● | ● | – | – | – | – | Inference platforms & gateways | |
| Neo4j LangSmith Evals | T1 | 4 | 1 | +3 | ● | ○ | ○ | ○ | ○ | ○ | ○ | Knowledge-graph platforms | |
| OpenHands Custom test harness | T1 | 4 | 1 | +3 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Agent IDEs & coding harnesses | |
| Amazon Neptune Analytics | T1 | 3 | 0 | +3 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Knowledge-graph platforms |
| Claude Code auto-memory | T1 | 3 | 0 | +3 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Claude Code memory mechanisms |
| CLAUDE.md | T1 | 3 | 0 | +3 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | File-backed / editor paradigms |
| Microsoft Copilot Memory | T1 | 3 | 0 | +3 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Platform-provider memory |
| v0 (Vercel) | T1 | 3 | 0 | +3 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses |
| DeepSeek R1 / V3 family LangSmith Evals | T1 | 6 | 4 | +2 | ● | ● | ● | ● | ○ | ○ | ○ | Foundation models (substrate reference) | |
| Amazon Q Developer Custom test harness | T1 | 3 | 1 | +2 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Agent IDEs & coding harnesses | |
| Claude Code (Anthropic) Custom test harness | T1 | 3 | 1 | +2 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Agent IDEs & coding harnesses | |
| LanceDB LangSmith Evals | T1 | 3 | 1 | +2 | ● | ○ | ○ | ○ | ○ | ○ | ○ | Vector-database infrastructure | |
| OpenAI Codex CLI Custom test harness | T1 | 3 | 1 | +2 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Agent IDEs & coding harnesses | |
| Cline (framework) | T1 | 2 | 0 | +2 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent frameworks (no first-party memory layer) |
| Continue.dev | T2 | 2 | 0 | +2 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses |
| Dify | T3 | 2 | 0 | +2 | – | – | – | – | – | – | – | Agent frameworks (no first-party memory layer) | |
| ElevenLabs Conversational AI (now ElevenAgents) | T1 | 2 | 0 | +2 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Voice agent platforms |
| Graphiti MCP Server (Zep) | T1 | 2 | 0 | +2 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Claude Code memory mechanisms |
| IBM ContextForge MCP Gateway | T3 | 2 | 0 | +2 | – | – | – | – | – | – | – | MCP gateways & registries | |
| Langflow | T3 | 2 | 0 | +2 | – | – | – | – | – | – | – | Agent frameworks (no first-party memory layer) | |
| Perplexity Deep Research | T1 | 2 | 0 | +2 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Use-case-specific agent harnesses |
| Sourcegraph Cody (agent) | T1 | 2 | 0 | +2 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses |
| LangGraph (orchestration) LangSmith Evals | T1 | 8 | 7 | +1 | ● | ● | ● | ● | ● | ● | ● | Agent frameworks (no first-party memory layer) | |
| CrewAI LangSmith Evals | T1 | 7 | 6 | +1 | ● | ● | ● | ● | ● | ● | ○ | Agent frameworks (no first-party memory layer) | |
| Alibaba Qwen 3 family LangSmith Evals | T1 | 5 | 4 | +1 | ● | ● | ● | ● | ○ | ○ | ○ | Foundation models (substrate reference) | |
| Datadog Bits AI Custom test harness | T1 | 3 | 2 | +1 | ○ | ○ | ○ | ○ | ● | ● | ○ | Use-case-specific agent harnesses | |
| Browser Use Custom test harness | T1 | 2 | 1 | +1 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Computer-use & desktop agents | |
| Datadog Agent Observability (LLM Observability) Human-in-loop | T3 | 2 | 1 | +1 | – | – | – | – | – | ● | – | Evaluation & observability platforms | |
| HoneyHive Human-in-loop | T3 | 2 | 1 | +1 | – | – | – | – | – | ● | – | Evaluation & observability platforms | |
| NVIDIA GR00T / Isaac Custom test harness | T1 | 2 | 1 | +1 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Robotics foundation models & agent stacks | |
| Agent Router (formerly Envoy AI Gateway) | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Inference platforms & gateways | |
| agentmemory (rohitg00) | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Coding-agent memory | |
| AGENTS.md | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | File-backed / editor paradigms |
| Arthur Engine | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Guardrails & AI security | |
| Bifrost (Maxim AI) | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Inference platforms & gateways | |
| Bolt.new (StackBlitz) — harness | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses |
| Browserbase | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Browser-agent memory |
| Claude for Chrome | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Computer-use & desktop agents |
| claude-mem | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Claude Code memory mechanisms |
| Cloudflare AI Gateway | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Inference platforms & gateways | |
| context-mode | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Coding-agent memory | |
| Cursor Rules | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | File-backed / editor paradigms |
| Docker MCP Catalog, Toolkit & Gateway | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | MCP gateways & registries | |
| Engram (DeepSeek) | T1 | 1 | 0 | +1 | – | – | – | – | – | – | – | Recent method papers — theorized, no distinct product | |
| GitHub Copilot Memory (agentic memory) | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Coding-agent memory | |
| GitHub Copilot Workspace | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses |
| Golem | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Durable execution & workflow engines | |
| Granola | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Voice-first / wearable AI memory |
| Hatchet | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Durable execution & workflow engines | |
| HubSpot Breeze AI Agents | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Use-case-specific agent harnesses |
| JetBrains AI Assistant | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses |
| Laminar | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Evaluation & observability platforms | |
| Lovable.dev — harness | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses |
| Lucidworks Conversational Q&A AI Agent | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Enterprise-search adjacencies |
| MCP Servers (Reference) | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent frameworks (no first-party memory layer) |
| Mem0 MCP (official) | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Claude Code memory mechanisms |
| Mem0 Security / OpenMemory | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Memory governance, privacy & safety |
| Merge Agent Handler | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Agent identity & auth | |
| Meta AI Memory | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Platform-provider memory |
| Microsoft Edge Copilot Mode + Journeys (now Copilot in Edge) | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Browser-agent memory |
| Model Context Protocol (MCP spec) | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent frameworks (no first-party memory layer) |
| NeoCognition | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Dedicated memory layers |
| NVIDIA NeMo Guardrails | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Guardrails & AI security | |
| NVIDIA Nemotron 3 | T1 | 1 | 0 | +1 | – | – | – | – | – | – | – | Recent method papers — theorized, no distinct product | |
| Official MCP Memory server | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Claude Code memory mechanisms |
| OpenAI ChatGPT Memory | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Platform-provider memory |
| OpenAI Deep Research | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Use-case-specific agent harnesses |
| OpenClaw memory | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | File-backed / editor paradigms | |
| Perplexity Comet | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Browser-agent memory |
| Physical Intelligence (π) | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Robotics foundation models & agent stacks |
| Project Mariner | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Computer-use & desktop agents |
| Sinequa | T1 | 1 | 0 | +1 | orphan | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Enterprise-search adjacencies |
| TrueFoundry AI Gateway | T3 | 1 | 0 | +1 | – | – | – | – | – | – | – | Inference platforms & gateways | |
| AutoGen / AG2 LangSmith Evals | T1 | 5 | 5 | 0 | ● | ● | ● | ○ | ● | ● | ○ | Agent frameworks (no first-party memory layer) | |
| Hugging Face LeRobot W&B Agent Eval | T1 | 2 | 2 | 0 | ○ | ○ | ● | ○ | ● | ○ | ○ | Robotics foundation models & agent stacks | |
| Letta / MemGPT Custom test harness | T1 | 2 | 2 | 0 | ○ | ○ | ○ | ○ | ● | ● | ○ | Dedicated memory layers | |
| Mem0 LangSmith Evals | T1 | 2 | 2 | 0 | ● | ○ | ○ | ○ | ● | ○ | ○ | Dedicated memory layers | |
| Salesforce Agentforce Custom test harness | T1 | 2 | 2 | 0 | ○ | ○ | ○ | ○ | ● | ● | ○ | Use-case-specific agent harnesses | |
| Character.ai Human-in-loop | T1 | 1 | 1 | 0 | ○ | ○ | ○ | ○ | ○ | ● | ○ | Vertical / domain-specific AI memory | |
| ChatGPT — Codex (cloud agent) Custom test harness | T1 | 1 | 1 | 0 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Agent IDEs & coding harnesses | |
| Cognee Custom test harness | T1 | 1 | 1 | 0 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Dedicated memory layers | |
| Google DeepMind Gemini Robotics Custom test harness | T1 | 1 | 1 | 0 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Robotics foundation models & agent stacks | |
| GraphRAG (Microsoft) Custom test harness | T1 | 1 | 1 | 0 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Retrieval-as-memory hybrids | |
| LinkedIn Cognitive Memory Agent Custom test harness | T1 | 1 | 1 | 0 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Platform-provider memory | |
| Microsoft Agent Framework Human-in-loop | T3 | 1 | 1 | 0 | – | – | – | – | – | ● | – | Agent frameworks (no first-party memory layer) | |
| OpenAI Operator Custom test harness | T1 | 1 | 1 | 0 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Computer-use & desktop agents | |
| Suki AI Human-in-loop | T1 | 1 | 1 | 0 | ○ | ○ | ○ | ○ | ○ | ● | ○ | Vertical / domain-specific AI memory | |
| Zep & Graphiti LangSmith Evals | T1 | 1 | 1 | 0 | ● | ○ | ○ | ○ | ○ | ○ | ○ | Dedicated memory layers | |
| Augment Code | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Coding-agent memory | |
| Charisma.ai | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Vertical / domain-specific AI memory | |
| Dia (Atlassian) | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Browser-agent memory | |
| Fellou | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Browser-agent memory | |
| Hindsight (Vectorize) | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Dedicated memory layers | |
| Interloom | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Dedicated memory layers | |
| Omi | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Voice-first / wearable AI memory | |
| Replit Agent v3 | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses | |
| Supermemory | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Dedicated memory layers | |
| Windsurf (now Devin Desktop) | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | Agent IDEs & coding harnesses | |
| Windsurf Rules & Memories (now Devin Desktop) | T1 | 0 | 0 | 0 | ○ | ○ | ○ | ○ | ○ | ○ | ○ | File-backed / editor paradigms | |
| Arize Phoenix LangSmith Evals | T1 | 3 | 4 | -1 | ● | ○ | ○ | ○ | ● | ● | ● | Evaluation & observability platforms | |
| Abridge Custom test harness | T1 | 1 | 2 | -1 | ○ | ○ | ○ | ○ | ● | ● | ○ | Vertical / domain-specific AI memory | |
| Cognition Devin v2 / Spec Mode Custom test harness | T1 | 1 | 2 | -1 | ○ | ○ | ○ | ○ | ● | ● | ○ | Agent IDEs & coding harnesses | |
| Decagon Custom test harness | T1 | 1 | 2 | -1 | ○ | ○ | ○ | ○ | ● | ● | ○ | Vertical / domain-specific AI memory | |
| Devin (Cognition) Custom test harness | T1 | 1 | 2 | -1 | ○ | ○ | ○ | ○ | ● | ● | ○ | Coding-agent memory | |
| Harvey Memory Custom test harness | T1 | 1 | 2 | -1 | ○ | ○ | ○ | ○ | ● | ● | ○ | Vertical / domain-specific AI memory | |
| Hippocratic AI Polaris Custom test harness | T1 | 1 | 2 | -1 | ○ | ○ | ○ | ○ | ● | ● | ○ | Vertical / domain-specific AI memory | |
| Sierra Custom test harness | T1 | 1 | 2 | -1 | ○ | ○ | ○ | ○ | ● | ● | ○ | Vertical / domain-specific AI memory | |
| Wayve GAIA-2 / GAIA-3 Custom test harness | T1 | 1 | 2 | -1 | ○ | ○ | ○ | ○ | ● | ○ | ● | Vertical / domain-specific AI memory | |
| Auth0 for AI Agents Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Agent identity & auth | |
| CAMEL Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Agent frameworks (no first-party memory layer) | |
| Cloudflare Workflows Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Durable execution & workflow engines | |
| DBOS Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Durable execution & workflow engines | |
| Descope Agentic Identity Hub Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Agent identity & auth | |
| Google Agent Development Kit (ADK) Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Agent frameworks (no first-party memory layer) | |
| HaluMem Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Memory benchmarks & evaluation | |
| Keycard Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Agent identity & auth | |
| Kiro Custom test harness | T1 | 0 | 1 | -1 | ○ | ○ | ○ | ○ | ● | ○ | ○ | Agent IDEs & coding harnesses | |
| OpenAI Evals W&B Agent Eval | T3 | 0 | 1 | -1 | – | – | ● | – | – | – | – | Evaluation & observability platforms | |
| Stytch Connected Apps Human-in-loop | T3 | 0 | 1 | -1 | – | – | – | – | – | ● | – | Agent identity & auth | |
| SWE-bench W&B Agent Eval | T3 | 0 | 1 | -1 | – | – | ● | – | – | – | – | Memory benchmarks & evaluation | |
| Braintrust Braintrust | T1 | 2 | 4 | -2 | ○ | ● | ○ | ○ | ● | ● | ● | Evaluation & observability platforms |