91.7% of catalogued products publish no peer-reviewed benchmark

The product × benchmark matrix is a 122 × 25 grid with 172 filled cells. Of the 3,050 possible product-benchmark cells, 94.4% are empty. Of the 172 filled cells, only 4 are independently verified (a score a third party has reproduced or posted on a neutral leaderboard); 115 are peer-reviewed paper claims, 26 are vendor blog or marketing numbers, and 27 are disputed (contested by a second source).

If you widen from the 122 products on the benchmark matrix to all 1,002 records in the catalog, 919 (91.7%) have no peer-reviewed benchmark score — meaning the product has shipped without a peer-reviewed result on any of the 25 benchmarks the catalog tracks (LoCoMo, LongMemEval, MTEB, GAIA, SWE-bench and the rest).

The implication: the agent-memory field's reliability story is built almost entirely on vendor self-reporting and paper claims, with effectively no public, neutral, reproducible benchmark coverage. A vendor that posts honest leaderboard scores would have category-leading transparency-as-positioning — there is nobody to compete with for that ground.