91.3% of catalogued products publish no peer-reviewed benchmark

The product × benchmark matrix in v6 is a 119 × 25 grid. The grid has 169 filled cells. Out of 119 × 25 = 2,975 possible product-benchmark cells, 94.3% are empty. Of the 169 filled cells, only 2 represent a score posted on a neutral leaderboard a third party can re-run; 111 are peer-reviewed paper claims, 52 are vendor blog/marketing numbers, and 4 are disputed (contested by a second source).

If you widen from the 119 products on the benchmark matrix to all 912 records in the catalog, 833 (91.3%) have zero benchmark cells filled at all — meaning the product has shipped without publishing a score on any of the 25 benchmarks the catalog tracks (LOCOMO, MTEB, GAIA, AgentBench, SWE-bench, BIG-bench, MMLU, and the rest).

The implication: the agent-memory field's reliability story is built almost entirely on vendor self-reporting and paper claims, with effectively no public, neutral, reproducible benchmark coverage. A vendor that posts honest leaderboard scores would have category-leading transparency-as-positioning — there is nobody to compete with for that ground.