{"family":"evals","name":"Evals & Observability","count":72,"license":"CC-BY-4.0","systems":[{"id":"agentbench--gh-thudm-agentbench","name":"AgentBench","url":"https://github.com/THUDM/AgentBench","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Benchmark","license":"Apache-2.0","latest_release":"last commit 2026-02-08","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/THUDM/AgentBench (3798★)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Multi-environment benchmark (ICLR'24) that evaluates LLMs acting as agents in multiple interactive environments, with code and data in the official repo.","cited_cells":12,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/agentbench--gh-thudm-agentbench"},{"id":"agentops--agentops-ai","name":"AgentOps","url":"https://www.agentops.ai/","section":"Memory observability & monitoring","tier":2,"status":"active","last_verified_at":"2026-10-22","type":"Operational metrics for agent memory","license":"MIT (SDK)","latest_release":"v0.4.21 (2025-08-29)","deployment":"SaaS cloud; SDK integration into agent code","pricing":"free tier; Pro/Enterprise pricing not publicly listed; contact sales","mcp_support":"via official adapter","a2a_support":"no data — searched agentops.ai, docs.agentops.ai, docs.agentops.ai/v2/integrations, README on AgentOps-AI/agentops; A2A protocol not advertised (CrewAI/AG2/LangGraph/Autogen integrations only)","backend_storage":"custom","llm_lock":"","github":"AgentOps-AI/agentops — ~5.8k stars","storage":"relational","retrieval":"exact-match","persistence":"long-term","update":"append-only","unit":"episode","governance":"auditable","conflict":"n/a","description":"When Mem0 is connected, gains Memory Operation Timeline, Search Analytics, Memory Growth tracking, Error Tracking per memory call. Standalone, records context at each step but doesn't analyse memory quality.","cited_cells":47,"third_party_cells":4,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/agentops--agentops-ai"},{"id":"androidworld--gh-google-research-android-world","name":"AndroidWorld","url":"https://github.com/google-research/android_world","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Mobile-OS agent benchmark — 116 real Android app tasks","license":"Apache-2.0","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"1.2k★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Google DeepMind's AndroidWorld — 116 hand-crafted tasks across 20 real Android apps (Calendar, Files, Markor, Tasks, etc.). Evaluates LLM agents controlling a real Android device via accessibility APIs. Distinct from AndroidEnv (RL-environment) — AndroidWorld is the benchmark suite. Released 2024; SOTA agent (M3A with GPT-4) at ~30% task success.","cited_cells":30,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/androidworld--gh-google-research-android-world"},{"id":"arize-ai--arize-com","name":"Arize AI","url":"https://arize.com/","section":"Evaluation & observability platforms","tier":1,"status":"active","last_verified_at":"2026-10-27","type":"ML + LLM observability platform","license":"proprietary; Python SDK Apache-2.0","latest_release":"v8.61.0 (2026-10-26) — arize","deployment":"Both (self-hosted + managed cloud) — AX SaaS; self-hosted on AX Enterprise only","pricing":"AX Free $0/mo (25k spans/mo); AX Pro $50/mo (50k spans/mo); AX Enterprise custom (SaaS or self-hosted); Phoenix OSS free","mcp_support":"yes — MCP servers and skills for coding agents to set up Arize AX","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"searched not found","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Production ML monitoring + LLM observability (Arize AX) — drift, eval, root-cause. Pre-LLM observability leader; now LLM-pivoted.","cited_cells":20,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/arize-ai--arize-com"},{"id":"arize-phoenix--phoenix-arize-com","name":"Arize Phoenix","url":"https://phoenix.arize.com/","section":"Evaluation & observability platforms","tier":1,"status":"active","last_verified_at":"2026-10-27","type":"OSS LLM tracing + eval","license":"Elastic-2.0","latest_release":"v20.26.1 (2026-10-25)","deployment":"searched not found","pricing":"searched not found","mcp_support":"yes — built-in remote MCP server at /mcp; legacy stdio @arizeai/phoenix-mcp in maintenance","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"Arize-ai/phoenix — ~12.0k stars, last commit 2026-10","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Arize AI's open-source LLM observability tool — OpenTelemetry-native LLM tracing, evals, datasets. Arize commercial offers Phoenix + Surveyor.","cited_cells":39,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/arize-phoenix--phoenix-arize-com"},{"id":"atari-100k--arxiv-1903-00374","name":"Atari 100k","url":"https://arxiv.org/abs/1903.00374","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Sample-efficient RL benchmark — 100k env steps on Atari","license":"GPL-2.0 (ALE)","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"2.3k★ C++ (ALE / atari-py)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Sample-efficiency benchmark protocol on Arcade Learning Environment (ALE) — agents limited to 100k environment steps (~2 hours of game-play) before evaluation. Introduced by SimPLe (Kaiser et al. 2019) and adopted by EfficientZero, IRIS, BBF, DreamerV3, Storm. Tests world-model / model-based methods where memory-as-imagination matters more than huge replay buffers.","cited_cells":28,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/atari-100k--arxiv-1903-00374"},{"id":"babilong--arxiv-2406-10149","name":"BABILong","url":"https://arxiv.org/abs/2406.10149","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Distributed multi-fact reasoning across long docs","license":"Apache-2.0","latest_release":"no releases","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"248★ Jupyter Notebook","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Extends 20 bAbI reasoning tasks (fact chaining, induction, deduction, counting, lists) by scattering relevant sentences across PG-19 book text. Scales to 10M tokens. Tests whether models can chain distributed facts, not merely retrieve a single needle.","cited_cells":29,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/babilong--arxiv-2406-10149"},{"id":"balrog--balrogai-com","name":"BALROG","url":"https://balrogai.com/","section":"Memory benchmarks & evaluation","tier":4,"status":"active","last_verified_at":"2026-10-28","type":"LLM/VLM agent benchmark on 6 challenging game environments","license":"MIT","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"200★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"BALROG (Benchmarking Agentic LLM and VLM Reasoning on Games) — 2024 benchmark suite covering NetHack, BabyAI, Crafter, MiniHack, TextWorld, Baba Is AI. Specifically targets long-horizon, partial-observability, exploration-heavy games where memory and planning matter. SOTA GPT-4o and Claude 3.5 Sonnet both score <25% on aggregate. Released by AI Safety Institute UK + Cambridge collaborators.","cited_cells":28,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/balrog--balrogai-com"},{"id":"beam-beyond-a-million-tokens-light--arxiv-2510-27246","name":"BEAM (Beyond a Million Tokens) + LIGHT","url":"https://arxiv.org/abs/2510.27246","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-23","type":"Benchmark","license":"MIT code; CC BY-SA 4.0 data","latest_release":"last commit 2026-08-31","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/mohammadtavakoli78/BEAM (164★)","storage":"none-trivial","retrieval":"none","persistence":"long-term","update":"read-only|append-only","unit":"turn|episode","governance":"n/a","conflict":"out-of-scope","description":"ICLR 2026 benchmark of 100 coherent conversations up to 10M tokens with 2,000 validated probes across ten memory abilities, including contradiction resolution and event ordering. It also proposes the LIGHT episodic/working/scratchpad memory method. Traction: ICLR 2026 paper; Mem0 publishes BEAM explainer and results","cited_cells":11,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/beam-beyond-a-million-tokens-light--arxiv-2510-27246"},{"id":"bench-tau-bench-bench--taubench-com","name":"τ-bench (tau-bench / τ²-bench)","url":"https://taubench.com/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Benchmark","license":"MIT","latest_release":"last commit 2026-10-29","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/sierra-research/tau2-bench (2354★)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Sierra's benchmark of whether agents can converse with simulated users, call tools and follow policy in domains such as retail, airline, telecom and banking. τ²-bench adds dual-control environments, and later releases add voice and knowledge tasks.","cited_cells":12,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/bench-tau-bench-bench--taubench-com"},{"id":"berkeley-function-calling-leaderboard-bfcl--gorilla-cs-berkeley-edu","name":"Berkeley Function Calling Leaderboard (BFCL)","url":"https://gorilla.cs.berkeley.edu/leaderboard.html","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Benchmark","license":"Apache-2.0","latest_release":"last commit 2026-03-23","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/ShishirPatil/gorilla (13063★)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Leaderboard that measures how accurately LLMs call functions (tools). It covers single-turn, multi-turn and agentic settings including web search and memory, plus hallucination and format sensitivity.","cited_cells":12,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/berkeley-function-calling-leaderboard-bfcl--gorilla-cs-berkeley-edu"},{"id":"braintrust--braintrust-dev","name":"Braintrust","url":"https://www.braintrust.dev/","section":"Evaluation & observability platforms","tier":1,"status":"active","last_verified_at":"2026-10-27","type":"Eval + observability for AI","license":"proprietary; SDKs open source","latest_release":"v0.48.1 (2026-10-26) — braintrust Python SDK","deployment":"Both (self-hosted + managed cloud) — hosted SaaS; on-prem deployment on Enterprise","pricing":"Starter $0/mo; Pro $249/mo; Enterprise custom","mcp_support":"yes — Braintrust MCP server (Connect with MCP quickstart)","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"searched not found","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Eval-first LLM platform — datasets, eval runs, prompt playground, tracing. Founded by ex-Impira (acquired by Dropbox) team. Used by Notion, Brex, AirOps.","cited_cells":43,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/braintrust--braintrust-dev"},{"id":"browsecomp--openai-com","name":"BrowseComp","url":"https://openai.com/index/browsecomp/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Benchmark","license":"MIT","latest_release":"last commit 2026-04-22","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/openai/simple-evals (4686★)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"OpenAI benchmark that measures whether browsing agents can find hard-to-locate information that may require visiting tens or hundreds of websites.","cited_cells":12,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/browsecomp--openai-com"},{"id":"claude-plays-pokemon-anthropic--twitch-tv","name":"Claude Plays Pokemon (Anthropic)","url":"https://www.twitch.tv/claudeplayspokemon","section":"Memory benchmarks & evaluation","tier":2,"status":"active","last_verified_at":"2026-10-28","type":"Anthropic agent demo — Claude 3.7/4 Sonnet plays Pokemon Red on Twitch","license":"","latest_release":"Twitch livestream ongoing; runs distinct per Claude model version","deployment":"not applicable — research demo","pricing":"not applicable — research demo / community project","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"","storage":"kv","retrieval":"extraction-pull","persistence":"session-only","update":"agent-controlled","unit":"episode","governance":"opaque","conflict":"agent","description":"Anthropic's public Twitch livestream of Claude (3.7 Sonnet → Claude 4 Opus) playing Pokemon Red. Agent uses screenshot vision + Game Boy controller actions + a scratchpad memory file the model can edit. Demonstrates long-horizon agentic capability publicly; ran continuously from Feb 2025; viewed by hundreds of thousands cumulatively; key data point for Anthropic's marketing around Claude as a long-horizon agent.","cited_cells":24,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/claude-plays-pokemon-anthropic--twitch-tv"},{"id":"confident-ai-deepeval--confident-ai-com","name":"Confident AI / DeepEval","url":"https://www.confident-ai.com/","section":"Evaluation & observability platforms","tier":2,"status":"active","last_verified_at":"2026-10-27","type":"LLM eval framework + cloud","license":"Apache-2.0 (DeepEval)","latest_release":"v4.3.1 (2026-10-16) — deepeval","deployment":"Both (self-hosted + managed cloud) — Confident AI cloud; dedicated on-prem on Enterprise; DeepEval runs locally","pricing":"Free $0; Starter $200/mo; Team $2,000/mo; Enterprise custom","mcp_support":"yes — Confident AI MCP server for Claude Code/Cursor; DeepEval MCP-use evaluation metrics","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"confident-ai/deepeval — ~19.1k stars, last commit 2026-10","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Open-source DeepEval framework + Confident AI cloud — pytest-style eval runs, hallucination/relevance/bias metrics, datasets.","cited_cells":23,"third_party_cells":3,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/confident-ai-deepeval--confident-ai-com"},{"id":"crafter--gh-danijar-crafter","name":"Crafter","url":"https://github.com/danijar/crafter","section":"Memory benchmarks & evaluation","tier":3,"status":"research-code","last_verified_at":"2026-10-28","type":"Minecraft-inspired 2D RL benchmark with 22 achievements","license":"MIT","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"500★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Danijar Hafner's 2D Minecraft-inspired benchmark — open-world survival with 22 hierarchical achievements (collect wood → make pickaxe → mine stone → ... → collect diamond). Procedurally-generated 64x64 grid world; tests long-horizon planning, exploration, credit assignment. Lightweight (single Python file) — designed as 'inner-loop Minecraft' for fast iteration. Used as flagship benchmark for DreamerV2/V3.","cited_cells":31,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/crafter--gh-danijar-crafter"},{"id":"datadog-agent-observability-llm-observability--datadoghq-com","name":"Datadog Agent Observability (LLM Observability)","url":"https://www.datadoghq.com/products/ai/agent-observability/","section":"Evaluation & observability platforms","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Product","license":"proprietary","latest_release":"searched not found","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Datadog's product for evaluating, tracing and monitoring AI agents, combining offline experiments on golden datasets with production tracing and troubleshooting. Its former LLM Observability URL now redirects here.","cited_cells":15,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/datadog-agent-observability-llm-observability--datadoghq-com"},{"id":"evo-memory--arxiv-2511-20857","name":"Evo-Memory","url":"https://arxiv.org/abs/2511.20857","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-23","type":"Benchmark","license":"unknown","latest_release":"searched not found","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"","storage":"none-trivial","retrieval":"none","persistence":"cross-session","update":"read-only","unit":"trajectory|episode","governance":"n/a","conflict":"out-of-scope","description":"Google streaming benchmark for test-time learning: agents must search, adapt and update memory across task sequences. It implements 10+ memory modules on 10 datasets and introduces the ExpRAG and ReMem baselines. Traction: Google (Ed Chi, Fernando Pereira et al.) authorship","cited_cells":6,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/evo-memory--arxiv-2511-20857"},{"id":"evomembench--alphaxiv-org","name":"EvoMemBench","url":"https://www.alphaxiv.org/abs/2605.18421","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-23","type":"Benchmark","license":"unspecified","latest_release":"last commit 2026-07-03","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/DSAIL-Memory/EvoMemBench (14★)","storage":"none-trivial","retrieval":"none","persistence":"cross-session|session","update":"read-only","unit":"episode|trajectory","governance":"n/a","conflict":"out-of-scope","description":"Benchmark that crosses memory scope (in-episode vs cross-episode) with content (knowledge vs execution) and compares 15 memory methods with long-context baselines under one protocol. Traction: Standardized comparison of 15 memory methods; HKUST(GZ) and others","cited_cells":11,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/evomembench--alphaxiv-org"},{"id":"gaia--huggingface-co","name":"GAIA","url":"https://huggingface.co/gaia-benchmark","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Benchmark","license":"searched not found","latest_release":"searched not found","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Benchmark for general AI assistants made of real-world questions that require reasoning, multimodality, web browsing and tool use, with a public leaderboard on Hugging Face. Traction: ~917 likes on the Hugging Face dataset","cited_cells":5,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/gaia--huggingface-co"},{"id":"galileo--galileo-ai","name":"Galileo","url":"https://www.galileo.ai/","section":"Evaluation & observability platforms","tier":1,"status":"acquired","last_verified_at":"2026-10-28","type":"Eval + observability + guardrails","license":"proprietary; Python SDK Apache-2.0","latest_release":"v2.7.0 (2026-10-14) — galileo","deployment":"Both (self-hosted + managed cloud) — SaaS, VPC or on-premises","pricing":"Splunk Agent Observability from $100/mo per 1.2M spans billed annually (includes 500M Luna tokens); free edition available","mcp_support":"yes — Agent Evals MCP server for IDEs such as Cursor and VS Code","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"rungalileo/galileo-python — ~22 stars, last commit 2026-10","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Galileo Evaluate / Observe / Protect — eval, tracing, guardrails; hallucination detection and ChainPoll metric. Acquired by Cisco in 2026 and now sold as Splunk Agent Observability.","cited_cells":21,"third_party_cells":4,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/galileo--galileo-ai"},{"id":"galileo-galileo-ai--galileo-ai","name":"Galileo (galileo.ai) (now Splunk Agent Observability)","url":"https://www.splunk.com/en_us/about-splunk/acquisitions/galileo.html","section":"Memory observability & monitoring","tier":1,"status":"acquired","last_verified_at":"2026-10-28","type":"Real-time intent / belief drift detection","license":"","latest_release":"","deployment":"SaaS cloud; Enterprise: VPC or on-prem","pricing":"Standalone pricing retired — galileo.ai/pricing now redirects to Splunk pricing; sold as Splunk Agent Observability","mcp_support":"Yes — Galileo MCP Server documented; integrates with traces, prompts, datasets","a2a_support":"Yes — A2A is listed under Integrations Overview (alongside CrewAI, Google ADK, LangChain, etc.)","backend_storage":"custom","llm_lock":"","github":"","storage":"vector","retrieval":"similarity","persistence":"cross-session","update":"append-only","unit":"episode","governance":"auditable","conflict":"n/a","description":"Acquired by Cisco (announced 2026-04-09, closed by 2026-05); now Splunk Agent Observability. Treats memory as first-class in multi-agent tracing. Luna-2 SLMs (3B / 8B) scan every interaction for intent drift and belief drift; 20+ checks at sub-200ms latency. Catches when agent A's view of the world splits from teammate B's. OpenTelemetry-compatible.","cited_cells":45,"third_party_cells":2,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/galileo-galileo-ai--galileo-ai"},{"id":"habitat-3-0-social-agents--aihabitat-org","name":"Habitat 3.0 (social agents)","url":"https://aihabitat.org/habitat3/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Embodied human-robot social interaction benchmark","license":"MIT (Habitat-Lab); CC-BY-NC (3D scenes)","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"2.6k★ Python/C++","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Meta AI's Habitat 3.0 — extends Habitat 2.0 with avatar humans, social tasks (navigate-while-following, rearrangement-while-not-blocking), and human-robot collaboration evaluation. Foundational for embodied social-agent research; supports both learned and scripted human avatars; SOTA realistic 3D scanned scenes (HM3D, Replica).","cited_cells":28,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/habitat-3-0-social-agents--aihabitat-org"},{"id":"halumem--arxiv-2511-03506","name":"HaluMem","url":"https://arxiv.org/abs/2511.03506","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-23","type":"Benchmark","license":"CC-BY-NC-ND-4.0","latest_release":"last commit 2026-09-03","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/MemTensor/HaluMem (178★)","storage":"none-trivial","retrieval":"none","persistence":"long-term","update":"read-only","unit":"fact|episode","governance":"n/a","conflict":"out-of-scope","description":"Benchmark that measures hallucination at each stage of an agent memory system (extraction, updating, QA) rather than only final answers; HaluMem-Medium/Long datasets have ~15k memory points and contexts over 1M tokens. Traction: From MemTensor and China Telecom; HF dataset IAAR-Shanghai/HaluMem; widely cited by memory vendors","cited_cells":11,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/halumem--arxiv-2511-03506"},{"id":"hanabi-learning-environment--gh-google-deepmind-hanabi-learning-environment","name":"Hanabi Learning Environment","url":"https://github.com/google-deepmind/hanabi-learning-environment","section":"Memory benchmarks & evaluation","tier":3,"status":"archived","last_verified_at":"2026-10-28","type":"Cooperative multi-agent partial-observability card-game benchmark","license":"Apache-2.0","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"700★ C++/Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"DeepMind's Hanabi Learning Environment — a benchmark for cooperative multi-agent agents under partial observability and theory-of-mind. Players see each other's cards but not their own; success requires modeling teammates' beliefs. Bard (Bowling et al. 2020) described Hanabi as 'a new frontier for AI research'. Used in OpenAI ad hoc team-play papers and DeepMind's Bayesian Action Decoder.","cited_cells":29,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/hanabi-learning-environment--gh-google-deepmind-hanabi-learning-environment"},{"id":"helicone--helicone-ai","name":"Helicone","url":"https://www.helicone.ai/","section":"Evaluation & observability platforms","tier":2,"status":"maintenance","last_verified_at":"2026-10-29","type":"LLM observability + gateway","license":"Apache-2.0 (Core)","latest_release":"v1.8.3 (2025-11-07) — @helicone/helpers","deployment":"Both (self-hosted + managed cloud) — Docker Compose self-host; Helm and on-prem for Enterprise","pricing":"Hobby free (10k requests); Pro $79/mo; Team $799/mo; Enterprise custom; plus usage-based pricing","mcp_support":"yes — @helicone/mcp server to query requests and sessions from MCP clients","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"Helicone/helicone — ~6.2k stars, last commit 2026-10","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Open-source LLM observability and AI gateway — one-line proxy integration captures every LLM call; sessions, custom properties, evals. Acquired by Mintlify in Mar 2026 and placed in maintenance mode; helicone.ai now states it is no longer part of Mintlify.","cited_cells":22,"third_party_cells":4,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/helicone--helicone-ai"},{"id":"helmet--arxiv-2410-02694","name":"HELMET","url":"https://arxiv.org/abs/2410.02694","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Holistic downstream-task long-context eval","license":"MIT","latest_release":"no-release no-release","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"214★ +2/mo Jupyter Notebook","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Princeton NLP. 7 task categories — RAG with real retrieval, cited generation, summarisation, re-ranking, classification, many-shot ICL, long-document QA — at lengths up to 128K. Model-based evaluation enables open-ended scoring.","cited_cells":30,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/helmet--arxiv-2410-02694"},{"id":"honeyhive--honeyhive-ai","name":"HoneyHive","url":"https://www.honeyhive.ai/","section":"Evaluation & observability platforms","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Product","license":"proprietary","latest_release":"searched not found","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Agent observability and evaluation platform for replaying traces step by step, debugging long-running multi-agent trajectories and evaluating live production traffic.","cited_cells":14,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/honeyhive--honeyhive-ai"},{"id":"humanloop--humanloop-com","name":"Humanloop","url":"https://humanloop.com/","section":"Evaluation & observability platforms","tier":2,"status":"archived","last_verified_at":"2026-10-29","type":"LLM eval + collab","license":"searched not found","latest_release":"v0.8.38 (2025-05-09) — humanloop SDK, final","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"humanloop/humanloop-python — ~12 stars, last commit 2025-05","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Collaborative LLM development platform — prompt editor, evals, tracing. UK; Index Ventures + Y Combinator backed. The team joined Anthropic in 2025 and the platform shut down on 2025-09-08.","cited_cells":13,"third_party_cells":4,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/humanloop--humanloop-com"},{"id":"iclr-2026-memagents-workshop--sites-google-com","name":"ICLR 2026 MemAgents Workshop","url":"https://sites.google.com/view/memagent-iclr26/","section":"Memory benchmarks & evaluation","tier":2,"status":"active","last_verified_at":"2026-10-23","type":"Workshop venue — RL + cognitive psychology + LLM memory","license":"","latest_release":"","deployment":"","pricing":"","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"","storage":"none-trivial","retrieval":"none","persistence":"n/a","update":"n/a","unit":"n/a","governance":"n/a","conflict":"out-of-scope","description":"Full-day hybrid workshop held April 27, 2026, Rio de Janeiro. Scope: episodic and semantic memory, working memory, parametric knowledge, KGs, vector DBs, retrieval pipelines, context management, long-context utilization, temporal credit assignment. Bridges RL, neuroscience, and cognitive psychology alongside the engineering track.","cited_cells":14,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/iclr-2026-memagents-workshop--sites-google-com"},{"id":"implicitmembench--arxiv-2604-08064","name":"ImplicitMemBench","url":"https://arxiv.org/abs/2604.08064","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Implicit / procedural memory benchmark","license":"","latest_release":"","deployment":"","pricing":"","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"300 items across three paradigms: procedural memory (one-shot skill retention after interference), priming (theme-driven response biases), classical conditioning (CS-US exposure shaping first decisions). Unified learning-interference-test protocol with first-attempt scoring to isolate automatized behavior — distinct from explicit-recall benchmarks.","cited_cells":16,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/implicitmembench--arxiv-2604-08064"},{"id":"infinitebench-bench--arxiv-2402-13718","name":"InfiniteBench (∞Bench)","url":"https://arxiv.org/abs/2402.13718","section":"Memory benchmarks & evaluation","tier":3,"status":"research-code","last_verified_at":"2026-10-28","type":"100K+ token realistic long-context comprehension","license":"MIT","latest_release":"no releases","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"382★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Tsinghua / OpenBMB. First benchmark with average input >100K tokens. 12 tasks across English + Chinese: long novel QA, code debugging, math in long context, summarisation, retrieval. Realistic documents (novels, code) rather than synthetic filler.","cited_cells":29,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/infinitebench-bench--arxiv-2402-13718"},{"id":"inspect-ai--inspect-aisi-org-uk","name":"Inspect AI","url":"https://inspect.aisi.org.uk/","section":"Evaluation & observability platforms","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Framework","license":"MIT","latest_release":"last commit 2026-10-29","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/UKGovernmentBEIS/inspect_ai (3128★)","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Open-source framework for frontier LLM evaluations from the UK AI Security Institute and Meridian Labs. It has composable datasets, agents, tools and scorers, 200+ prebuilt evals including agentic tasks, and a log viewer.","cited_cells":14,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/inspect-ai--inspect-aisi-org-uk"},{"id":"laminar--laminar-sh","name":"Laminar","url":"https://laminar.sh/","section":"Evaluation & observability platforms","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Product","license":"Apache-2.0","latest_release":"last commit 2026-10-16","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/lmnr-ai/lmnr (3460★)","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Open-source agent observability platform that captures LLM calls, tool calls, sub-agents, costs and tokens. Its Signals feature clusters agent failures automatically, and it also runs evals.","cited_cells":16,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/laminar--laminar-sh"},{"id":"langfuse--langfuse-com","name":"Langfuse","url":"https://langfuse.com/","section":"Memory observability & monitoring","tier":1,"status":"acquired","last_verified_at":"2026-10-29","type":"Memory ops as named spans + Agent Graphs","license":"MIT (core; ee/ directories under separate enterprise license)","latest_release":"v4.27.0 (Sep 1; page shows no year, presumably 2026)","deployment":"SaaS cloud (Berlin + SF) or self-hosted Docker/K8s; ClickHouse native integration post-acquisition","pricing":"free: 50K observations/mo; Pro: $59/mo; Team: $199/mo; Enterprise: custom","mcp_support":"via official adapter — langfuse-mcp","a2a_support":"no data — searched langfuse.com, langfuse.com/docs, langfuse.com/integrations; A2A protocol not advertised (extensive framework integrations, MCP server present, OTel-native)","backend_storage":"Postgres + ClickHouse","llm_lock":"","github":"langfuse/langfuse — ~36.1k stars, last release 2026-09","storage":"relational","retrieval":"exact-match","persistence":"long-term","update":"append-only","unit":"episode","governance":"auditable","conflict":"n/a","description":"Acquired by ClickHouse (2026-01-16); product and MIT-licensed repo continue under the Langfuse name. Memory module reads/writes captured as named spans. Trace Log View concatenates every agent step including memory ops. Agent Graphs (GA 2025) infer graph structure from observation nesting; session-level replay tracks how memory state evolves.","cited_cells":51,"third_party_cells":8,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/langfuse--langfuse-com"},{"id":"langsmith--langchain-com","name":"LangSmith","url":"https://www.langchain.com/langsmith","section":"Memory observability & monitoring","tier":1,"status":"active","last_verified_at":"2026-10-23","type":"Memory mutations as distinct span types","license":"","latest_release":"SDK v0.14.11 (2026-10-16) — platform itself proprietary","deployment":"SaaS cloud (US); Enterprise: dedicated tenant","pricing":"Developer: free (5K traces/mo); Plus: $39/seat/mo (10K traces); Enterprise: custom (SSO/SAML; dedicated support)","mcp_support":"via LangChain MCP adapters","a2a_support":"no data — searched langchain.com/langsmith, docs.smith.langchain.com, docs.smith.langchain.com/observability/concepts; A2A protocol not advertised (OTel-based traces, webhook automations)","backend_storage":"custom (LangChain-managed)","llm_lock":"","github":"","storage":"relational","retrieval":"exact-match","persistence":"long-term","update":"append-only","unit":"episode","governance":"auditable","conflict":"n/a","description":"Memory reads, vector DB retrievals, state changes are distinct span types in traces. RAG eval separates retrieval quality (context precision) from generation quality (faithfulness). Dataset versioning guards against eval drift.","cited_cells":45,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/langsmith--langchain-com"},{"id":"locomo--arxiv-2402-17753","name":"LoCoMo","url":"https://arxiv.org/abs/2402.17753","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-23","type":"Very long-term multi-session dialogue memory","license":"MIT (paper code release)","latest_release":"no-release no-release","deployment":"not applicable — evaluation harness; run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"39★ Jupyter Notebook","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Conversations spanning up to 35 sessions / 300 turns / ~9K tokens, built from LLM-agent personas grounded on temporal event graphs and verified by human annotators. Tests QA (single-hop, multi-hop, temporal, commonsense, adversarial), event-graph summarisation, multimodal dialogue.","cited_cells":29,"third_party_cells":6,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/locomo--arxiv-2402-17753"},{"id":"longbench--arxiv-2308-14508","name":"LongBench","url":"https://arxiv.org/abs/2308.14508","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Bilingual multitask long-context suite","license":"MIT","latest_release":"no releases","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"1.2k★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Tsinghua THUDM. 21 datasets across 6 categories (single-doc QA, multi-doc QA, summarisation, few-shot ICL, synthetic, code completion) in English + Chinese. ~6.7K words English / 13.4K chars Chinese average. v2 at ACL 2025 with harder, longer instances.","cited_cells":29,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/longbench--arxiv-2308-14508"},{"id":"longmemeval--arxiv-2410-10813","name":"LongMemEval","url":"https://arxiv.org/abs/2410.10813","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-23","type":"Long-term chat-assistant memory across sessions","license":"MIT","latest_release":"no releases","deployment":"not applicable — evaluation harness; run locally against model API","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"740★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"5 core memory abilities — info extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention — via 500 curated questions in synthetic chat histories. Two scales: LongMemEval-S (115K tokens), LongMemEval-M (up to 1.5M).","cited_cells":28,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/longmemeval--arxiv-2410-10813"},{"id":"membench--arxiv-2506-21605","name":"MemBench","url":"https://arxiv.org/abs/2506.21605","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Multi-dim agent memory mechanism comparison","license":"","latest_release":"","deployment":"","pricing":"","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Renmin University. Evaluates LLM-agent memory mechanisms across three axes — effectiveness (accuracy), efficiency (LLM call count), capacity (degradation as memory grows). Distinguishes factual vs reflective memory, participation vs observation scenarios.","cited_cells":15,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/membench--arxiv-2506-21605"},{"id":"memconflict--arxiv-2605-20926","name":"MemConflict","url":"https://arxiv.org/abs/2605.20926","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-23","type":"Benchmark","license":"unspecified","latest_release":"last commit 2026-06-27","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/TaoZhen1110/MemConflict (13★)","storage":"none-trivial","retrieval":"none","persistence":"cross-session","update":"read-only","unit":"turn|fact","governance":"n/a","conflict":"out-of-scope","description":"Diagnostic benchmark covering dynamic, static and conditional memory conflicts, with black-box answer and white-box retrieval/ranking metrics; it evaluates A-Mem, LangMem, Letta, MemOS, Mem0 and Memobase. Traction: Head-to-head evaluation of six named memory systems; Renmin Univ + MemTensor","cited_cells":10,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/memconflict--arxiv-2605-20926"},{"id":"memevobench--arxiv-2604-15774","name":"MemEvoBench","url":"https://arxiv.org/abs/2604.15774","section":"Memory benchmarks & evaluation","tier":4,"status":"active","last_verified_at":"2026-10-28","type":"Adversarial memory-evolution safety benchmark","license":"","latest_release":"","deployment":"","pricing":"","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"QA tasks across 7 domains / 36 risk types plus workflow tasks adapted from 20 Agent-SafetyBench environments with noisy tool returns; both use mixed benign-and-misleading memory pools in multi-round interactions. Targets memory-evolution safety, not recall accuracy — first benchmark specifically targeting adversarial degradation of evolving memory.","cited_cells":15,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/memevobench--arxiv-2604-15774"},{"id":"memoryagentbench--arxiv-2507-05257","name":"MemoryAgentBench","url":"https://arxiv.org/abs/2507.05257","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"4-competency agent memory benchmark","license":"Unlicensed","latest_release":"no releases","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use (unlicensed GitHub)","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"323★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"HUST. Reconstructs existing datasets into incremental multi-turn dialogue chunks fed in time order. Plus EventQA (accurate retrieval) and FactConsolidation (selective forgetting). Targets retrieval, test-time learning, long-range understanding, selective forgetting.","cited_cells":26,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/memoryagentbench--arxiv-2507-05257"},{"id":"memoryarena--arxiv-2602-16313","name":"MemoryArena","url":"https://arxiv.org/abs/2602.16313","section":"Memory benchmarks & evaluation","tier":4,"status":"active","last_verified_at":"2026-10-28","type":"Multi-session agentic memory eval gym","license":"","latest_release":"","deployment":"","pricing":"","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Stanford Digital Economy Lab et al. Benchmarks memory in realistic Memory-Agent-Environment loops where tasks span multiple sessions and subtasks are explicitly interdependent — agents must distill prior-session experiences into memory and use that memory to act later. Exposes the gap between conversational memory benchmarks (LoCoMo) and true agentic memory.","cited_cells":15,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/memoryarena--arxiv-2602-16313"},{"id":"mind2web--osu-nlp-group-github-io","name":"Mind2Web","url":"https://osu-nlp-group.github.io/Mind2Web/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Benchmark","license":"MIT (code); dataset CC-BY-4.0","latest_release":"last commit 2025-11-05","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/OSU-NLP-Group/Mind2Web (1036★)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Dataset and benchmark for generalist web agents with 2,350 language-instruction tasks on 137 real websites across 31 domains.","cited_cells":12,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/mind2web--osu-nlp-group-github-io"},{"id":"minedojo--minedojo-org","name":"MineDojo","url":"https://minedojo.org/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Open-ended Minecraft agent benchmark + knowledge base","license":"MIT (env wrapper); Mojang EULA on Minecraft","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"2.4k★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"NVIDIA + Caltech open-ended embodied-agent benchmark on Minecraft. 3000+ programmatic + creative tasks across survival / harvest / tech-tree / combat / creative. Includes an internet-scale knowledge base (730K YouTube videos, 7K wiki pages, 340K Reddit posts) for grounding. Released at NeurIPS 2022 (outstanding paper award).","cited_cells":28,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/minedojo--minedojo-org"},{"id":"minerl-diamond-challenge--minerl-io","name":"MineRL Diamond Challenge","url":"https://minerl.io/","section":"Memory benchmarks & evaluation","tier":2,"status":"active","last_verified_at":"2026-05-14","type":"Minecraft RL benchmark — obtain a diamond","license":"MIT","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"~700★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"MineRL is a research project / competition series using Minecraft as a long-horizon RL environment. The flagship Diamond Challenge tasks an agent with obtaining a diamond from raw world — requires deep tech-tree planning across thousands of steps. DreamerV3 (DeepMind, 2023-12) famously became the first to solve it from scratch without human data.","cited_cells":28,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/minerl-diamond-challenge--minerl-io"},{"id":"nethack-learning-environment-nle--gh-heiner-nle","name":"NetHack Learning Environment (NLE)","url":"https://github.com/heiner/nle","section":"Memory benchmarks & evaluation","tier":2,"status":"active","last_verified_at":"2026-05-14","type":"Roguelike RL benchmark — NetHack as agent environment","license":"NetHack General Public License","latest_release":"v0.9.x series (active as of 2024)","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"1.0k★ Python/C","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"DeepMind / FAIR's NetHack Learning Environment — a Gym-compatible interface to NetHack, the canonical roguelike. Massive procedural state space, permadeath, deep item / skill interactions, ASCII partial-observability. Released at NeurIPS 2020; canonical long-horizon partial-observability RL benchmark.","cited_cells":30,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/nethack-learning-environment-nle--gh-heiner-nle"},{"id":"niah-needle-in-a-haystack--gh-gkamradt-llmtest-needleinahaystack","name":"NIAH (Needle in a Haystack)","url":"https://github.com/gkamradt/LLMTest_NeedleInAHaystack","section":"Memory benchmarks & evaluation","tier":4,"status":"active","last_verified_at":"2026-05-14","type":"Single-fact verbatim retrieval from long context","license":"MIT","latest_release":"no releases","deployment":"not applicable — run locally","pricing":"not applicable — open source; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"2.3k★ Jupyter Notebook","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Places a known sentence (\"needle\") at varying positions inside a long irrelevant document (\"haystack\") and prompts the model to retrieve it. Tests context lengths from 2K to model max. Greg Kamradt, 2023, no formal venue.","cited_cells":25,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/niah-needle-in-a-haystack--gh-gkamradt-llmtest-needleinahaystack"},{"id":"openai-evals--gh-openai-evals","name":"OpenAI Evals","url":"https://github.com/openai/evals","section":"Evaluation & observability platforms","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Framework","license":"MIT","latest_release":"last commit 2026-04-14","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/openai/evals (19675★)","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"OpenAI's open-source framework and registry of benchmarks for evaluating LLMs and LLM systems.","cited_cells":13,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/openai-evals--gh-openai-evals"},{"id":"openai-gym-retro--gh-openai-retro","name":"OpenAI Gym Retro","url":"https://github.com/openai/retro","section":"Memory benchmarks & evaluation","tier":3,"status":"archived","last_verified_at":"2026-10-28","type":"Retro arcade RL benchmark — Sega / NES / SNES game integration","license":"MIT","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"3.5k★ Python/C","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"OpenAI's Gym Retro — wraps thousands of classic arcade games (Sega Genesis, NES, SNES, Atari 2600, Game Boy) as Gym RL environments via libretro emulators. Released 2018; powered the OpenAI Retro Contest where agents had to generalise from train-set games to held-out levels of Sonic. Largely supplanted by Atari 100k / Procgen / etc. but still the most-popular path for adding arbitrary retro games to RL pipelines.","cited_cells":30,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/openai-gym-retro--gh-openai-retro"},{"id":"openinference--gh-arize-ai-openinference","name":"OpenInference","url":"https://github.com/Arize-ai/openinference","section":"Evaluation & observability platforms","tier":2,"status":"active","last_verified_at":"2026-10-27","type":"OTel semantic conventions for LLMs","license":"Apache-2.0","latest_release":"v0.1.42 (2026-10-23) — openinference-semantic-conventions","deployment":"Self-hosted only — library","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"Arize-ai/openinference — ~1.3k stars, last commit 2026-10","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Arize-led open OTel semantic conventions for LLM/agent tracing — auto-instrumentation for OpenAI/Anthropic/LangChain/LlamaIndex etc.","cited_cells":19,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/openinference--gh-arize-ai-openinference"},{"id":"openllmetry-traceloop--gh-traceloop-openllmetry","name":"OpenLLMetry (Traceloop)","url":"https://github.com/traceloop/openllmetry","section":"Evaluation & observability platforms","tier":2,"status":"active","last_verified_at":"2026-10-27","type":"OTel auto-instrumentation for LLMs","license":"Apache-2.0","latest_release":"v0.62.4 (2026-09-29) — traceloop-sdk","deployment":"Self-hosted only — library","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"traceloop/openllmetry — ~7.5k stars, last commit 2026-10","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Traceloop's open-source OpenTelemetry auto-instrumentation for LLM workflows — works with any OTel-compatible backend (Datadog, New Relic, Grafana, etc.).","cited_cells":21,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/openllmetry-traceloop--gh-traceloop-openllmetry"},{"id":"osrs-bench-old-school-runescape-agent-benchmark--gh-grahamannett-osrs-bench","name":"OSRS Bench (Old School RuneScape agent benchmark)","url":"https://github.com/grahamannett/osrs-bench","section":"Memory benchmarks & evaluation","tier":4,"status":"inactive","last_verified_at":"2026-05-14","type":"Long-horizon MMO agent benchmark — Old School RuneScape","license":"MIT (assumed — community repo, no explicit LICENSE file checked)","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"small community repo (<50★), Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Community benchmark suite for LLM agents playing Old School RuneScape, a sandbox MMORPG with deep partial-observability, skill-grinding loops, and economy interaction. Tests long-horizon planning (hour-to-day quests), tool inventory management, and visual UI comprehension. Community-built; not affiliated with Jagex.","cited_cells":24,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/osrs-bench-old-school-runescape-agent-benchmark--gh-grahamannett-osrs-bench"},{"id":"osworld--os-world-github-io","name":"OSWorld","url":"https://os-world.github.io/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Desktop OS agent benchmark — 369 tasks across Ubuntu / Windows / macOS","license":"Apache-2.0","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"1.7k★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"HKU NLP + Salesforce's OSWorld — full-desktop agent benchmark with 369 tasks spanning real applications (LibreOffice, GIMP, Chrome, VS Code, Thunderbird, Files) across Ubuntu, Windows, and macOS. Tasks require multi-application workflows; evaluation via post-task state inspection (not just screenshot match). Released NeurIPS 2024.","cited_cells":28,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/osworld--os-world-github-io"},{"id":"patronus-ai--patronus-ai","name":"Patronus AI","url":"https://www.patronus.ai/","section":"Evaluation & observability platforms","tier":2,"status":"archived","last_verified_at":"2026-10-29","type":"Enterprise eval + guardrails","license":"proprietary; Python SDK MIT","latest_release":"v0.1.26 (2026-09-17) — patronus","deployment":"searched not found","pricing":"Contact sales — no public price list","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"searched not found","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Enterprise eval and guardrails platform — Lynx hallucination detector (open weights), Glider judge model, Polyglot benchmark. Built by ex-Meta researchers. The evaluation platform (API, SDKs, evaluators, Lynx, GLIDER, Percival) has been discontinued per its docs (checked 2026-10-29); Patronus AI now operates as a behavioral research lab building training data and simulated environments for agents.","cited_cells":13,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/patronus-ai--patronus-ai"},{"id":"pokemon-red-benchmark-speedrun-completion--gh-pwhiddy-pokemonredexperiments","name":"Pokemon Red benchmark (speedrun / completion)","url":"https://github.com/PWhiddy/PokemonRedExperiments","section":"Memory benchmarks & evaluation","tier":4,"status":"active","last_verified_at":"2026-10-28","type":"Pokemon Red completion benchmark — long-horizon JRPG agent task","license":"MIT (Whiddy repo)","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"7.6k★ Python (PokemonRedExperiments)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Pokemon Red has emerged as a community + academic benchmark for long-horizon agentic AI. Notable instances: Peter Whiddy's PokemonRedExperiments (RL via PPO + curiosity, viral YouTube series 2023); Anthropic's ClaudePlaysPokemon Twitch stream (2025-02); Google's GeminiPlaysPokemon (2025); MorphLabs PokemonRedRL. The benchmark protocol varies: badge count, gym progression, full Elite-Four completion. Common across all: ~5k-20k+ step horizon, sparse rewards, deep partial-observability.","cited_cells":30,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/pokemon-red-benchmark-speedrun-completion--gh-pwhiddy-pokemonredexperiments"},{"id":"procgen-benchmark-suite--gh-openai-procgen","name":"PROCGEN benchmark suite","url":"https://github.com/openai/procgen","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Procedurally-generated RL game suite — generalisation eval","license":"MIT","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"980★ Python/C","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"OpenAI's 16-game procedurally-generated benchmark for evaluating RL generalisation. Unlike Atari (fixed levels memorisable), every PROCGEN episode uses a different procedural seed → test-set generalisation is the headline metric. Released at ICML 2020. Standard benchmark for studying generalisation, exploration, transfer.","cited_cells":30,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/procgen-benchmark-suite--gh-openai-procgen"},{"id":"promptlayer--promptlayer-com","name":"PromptLayer","url":"https://www.promptlayer.com/","section":"Evaluation & observability platforms","tier":3,"status":"active","last_verified_at":"2026-10-27","type":"Prompt management + tracing","license":"proprietary; Python SDK Apache-2.0","latest_release":"v1.5.16 (2026-08-19) — promptlayer","deployment":"Both (self-hosted + managed cloud) — self-hosting on Enterprise plan (GCP, AWS, Azure)","pricing":"Free $0 (2.5k requests/mo); Pro $49/mo; Team $500/mo (100k+ requests); Enterprise custom","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"MagnivOrg/prompt-layer-library — ~0.8k stars, last commit 2026-08","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Prompt versioning + tracing platform — non-eng-friendly prompt editor; early entrant pre-LLM-ops wave.","cited_cells":20,"third_party_cells":2,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/promptlayer--promptlayer-com"},{"id":"pydantic-logfire--pydantic-dev","name":"Pydantic Logfire","url":"https://pydantic.dev/logfire","section":"Evaluation & observability platforms","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Product","license":"proprietary; Python SDK MIT","latest_release":"last commit 2026-10-29","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/pydantic/logfire (4526★)","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"OpenTelemetry-based AI observability platform that traces model calls, agent steps, API requests and database queries in one view. SDKs are available for Python, JS/TS and Rust.","cited_cells":22,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/pydantic-logfire--pydantic-dev"},{"id":"ragas--gh-explodinggradients-ragas","name":"Ragas","url":"https://github.com/vibrantlabsai/ragas","section":"Evaluation & observability platforms","tier":2,"status":"active","last_verified_at":"2026-10-28","type":"RAG eval metrics","license":"Apache-2.0","latest_release":"v0.4.3 (2026-01-13) — ragas","deployment":"Self-hosted only — library","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"vibrantlabsai/ragas — ~16.2k stars, last commit 2026-02","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Open-source eval framework for RAG pipelines — faithfulness, answer-relevance, context-precision, context-recall. Widely used as a baseline RAG eval kit.","cited_cells":18,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/ragas--gh-explodinggradients-ragas"},{"id":"ratine--gh-goweft-ratine","name":"Ratine","url":"https://github.com/goweft/ratine","section":"Memory observability & monitoring","tier":2,"status":"active","last_verified_at":"2026-10-22","type":"Offline scanner — memory poisoning detector","license":"MIT","latest_release":"v0.1.0 2026-04-02","deployment":"local CLI — Python; pip install; no server required","pricing":"free / open source (MIT); no paid tier","mcp_support":"Yes — built for MCP tool-server ecosystem (sibling tools heddle/loom are MCP runtimes); detects agent memory poisoning across MCP sessions","a2a_support":"no data — searched github.com/goweft/ratine README; single-purpose memory-poisoning detector for MCP tool servers, no A2A protocol","backend_storage":"searched not found","llm_lock":"","github":"1★ Python","storage":"file","retrieval":"extraction-pull","persistence":"cross-session","update":"read-only","unit":"fact","governance":"auditable","conflict":"n/a","description":"Only tool found that scans the persistent memory layer on disk rather than runtime tracing. Detects injected instructions, obfuscated payloads (zero-width Unicode, base64, homoglyphs, hex), C2-pattern URLs, credential leakage. ratine diff compares snapshots for belief drift.","cited_cells":42,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/ratine--gh-goweft-ratine"},{"id":"ruler--arxiv-2404-06654","name":"RULER","url":"https://arxiv.org/abs/2404.06654","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Configurable multi-task long-context stress test","license":"Apache-2.0","latest_release":"no releases","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"1.5k★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"NVIDIA. Extends NIAH with multi-needle retrieval, multi-hop entity tracing, aggregation, QA — configurable for arbitrary length / difficulty. Evaluated 17 long-context models on 13 tasks.","cited_cells":29,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/ruler--arxiv-2404-06654"},{"id":"smartplay--gh-microsoft-smartplay","name":"SmartPlay","url":"https://github.com/Microsoft/SmartPlay","section":"Memory benchmarks & evaluation","tier":3,"status":"archived","last_verified_at":"2026-10-28","type":"LLM-agent benchmark across 6 games — Microsoft Research","license":"MIT","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"170★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Microsoft Research's SmartPlay — LLM-agent benchmark on 6 games (Bandits, Rock-Paper-Scissors, Tower of Hanoi, MessengerEnv, Crafter, Minecraft). Tests 9 distinct capability axes including object understanding, planning, generalisation, error-handling, theory-of-mind. ICLR 2024. Bridges narrow-task benchmarks (MMLU) and full-agent benchmarks (AgentBench).","cited_cells":29,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/smartplay--gh-microsoft-smartplay"},{"id":"stale-cupmem--arxiv-2605-06527","name":"STALE (+ CUPMEM)","url":"https://arxiv.org/abs/2605.06527","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-23","type":"Benchmark","license":"unknown","latest_release":"searched not found","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"","storage":"none-trivial|kv","retrieval":"none|hybrid","persistence":"long-term","update":"read-only|consolidation","unit":"fact","governance":"n/a","conflict":"llm-arbitrate|out-of-scope","description":"Conflict/update benchmark of 400 scenarios with 1,200 queries (up to ~150K tokens) testing whether agents notice that later facts implicitly invalidate earlier memories; the best model scores 55.2%. The CUPMEM prototype adds write-time state adjudication. Traction: Directly targets implicit memory conflict; HKUST/Wuhan/CUHK","cited_cells":6,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/stale-cupmem--arxiv-2605-06527"},{"id":"swe-bench--swebench-com","name":"SWE-bench","url":"https://www.swebench.com/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Benchmark","license":"MIT","latest_release":"last commit 2026-09-02","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/SWE-bench/SWE-bench (6117★)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Benchmark of real GitHub issues that coding agents must resolve with a patch, with official leaderboards for Verified (500 human-filtered instances), Lite, Multimodal and Multilingual.","cited_cells":13,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/swe-bench--swebench-com"},{"id":"terminal-bench--tbench-ai","name":"Terminal-Bench","url":"https://www.tbench.ai/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-29","type":"Benchmark","license":"Apache-2.0","latest_release":"last commit 2026-07-11","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"searched not found","llm_lock":"searched not found","github":"https://github.com/harbor-framework/terminal-bench-1 (2613★)","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Benchmark of agents completing hard tasks in a terminal environment, now at version 4.0, with a public leaderboard of model and agent pairs. It is hosted by Stanford, Harbor and the Laude Institute.","cited_cells":12,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/terminal-bench--tbench-ai"},{"id":"textworld--gh-microsoft-textworld","name":"TextWorld","url":"https://github.com/microsoft/TextWorld","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Text-adventure RL benchmark — procedurally-generated IF games","license":"MIT","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"1.3k★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"Microsoft Research's TextWorld — a sandbox for training and evaluating RL agents on text-based games. Procedurally generates interactive-fiction games of tunable difficulty (map size, quest length, ingredients) with full ground-truth knowledge graphs. Used as the basis for the FirstTextWorld and TextWorld Challenge competitions.","cited_cells":30,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/textworld--gh-microsoft-textworld"},{"id":"truera--truera-com","name":"TruEra","url":"https://truera.com/","section":"Evaluation & observability platforms","tier":2,"status":"acquired","last_verified_at":"2026-10-29","type":"ML observability + RAG triad","license":"searched not found","latest_release":"v12.6.1 (2024-03-04) — truera (Python SDK)","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"searched not found","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"ML observability with the TruLens open-source LLM eval framework — the 'RAG Triad' (context-relevance, groundedness, answer-relevance) metrics. Acquired by Snowflake 2024-05.","cited_cells":9,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/truera--truera-com"},{"id":"trulens--gh-truera-trulens","name":"TruLens","url":"https://github.com/truera/trulens","section":"Evaluation & observability platforms","tier":2,"status":"active","last_verified_at":"2026-10-29","type":"LLM eval framework","license":"MIT","latest_release":"v2.16.0 (2026-10-19) — trulens","deployment":"Self-hosted only — library","pricing":"searched not found","mcp_support":"yes — instruments MCP tool calls via the MCP span type (tool name, arguments, output, latency)","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"truera/trulens — ~3.6k stars, last commit 2026-10","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Open-source LLM/RAG evaluation framework — feedback functions, the RAG Triad. Originally TruEra; now Snowflake-stewarded.","cited_cells":20,"third_party_cells":1,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/trulens--gh-truera-trulens"},{"id":"vellum-ai--vellum-ai","name":"Vellum AI","url":"https://www.vellum.ai/","section":"Evaluation & observability platforms","tier":2,"status":"dead","last_verified_at":"2026-10-29","type":"LLM dev platform (build + eval + deploy)","license":"proprietary; Python SDK MIT","latest_release":"v1.14.8 (2026-10-02) — vellum-ai (Python SDK)","deployment":"searched not found","pricing":"searched not found","mcp_support":"searched not found","a2a_support":"searched not found","backend_storage":"","llm_lock":"","github":"searched not found","storage":"","retrieval":"","persistence":"","update":"","unit":"","governance":"","conflict":"","description":"Originally an enterprise LLM development platform (prompts, workflows, evals). It pivoted to a personal AI assistant (open-source vellum-assistant plus Vellum Cloud). Vellum announced its hosted Personal Assistant Platform will be shut down on 2026-11-02 and that it will no longer actively maintain the open-source repo.","cited_cells":13,"third_party_cells":2,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/vellum-ai--vellum-ai"},{"id":"webarena--webarena-dev","name":"WebArena","url":"https://webarena.dev/","section":"Memory benchmarks & evaluation","tier":3,"status":"active","last_verified_at":"2026-10-28","type":"Web-agent benchmark — 812 tasks across 6 self-hosted sites","license":"Apache-2.0","latest_release":"","deployment":"not applicable — run locally","pricing":"not applicable — open benchmark; free to use","mcp_support":"","a2a_support":"","backend_storage":"","llm_lock":"","github":"1.2k★ Python","storage":"n/a","retrieval":"n/a","persistence":"n/a","update":"read-only","unit":"n/a","governance":"n/a","conflict":"n/a","description":"CMU's WebArena — 812 hand-crafted web tasks across 6 self-hosted reproductions of real websites (Reddit-clone Postmill, GitLab, OneStopShop, CMS, Map, Wikipedia). Released ICLR 2024. Distinct from Mind2Web (real-website screenshots) — WebArena is fully reproducible with Docker-deployable backends. Spawned VisualWebArena, WorkArena, and dozens of agent benchmarks.","cited_cells":28,"third_party_cells":0,"page":"https://mrpeppersdev.github.io/agent-infrastructure-landscape/systems/webarena--webarena-dev"}]}