# AI labs report benchmark scores 6.4% higher than independent testers, on average — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/68579f0f8e1dd3ad

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-19 09:47 ingested 2026-10-20 02:39

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1x9skrx/ai_labs_report_benchmark_scores_64_higher_than/)

## retained text



```
I've been building [whatmodel.sh](http://whatmodel.sh), an open source AI leaderboard, and wanted to see how a labs' own benchmark scores compare with independent testers.

Using 1,629 published benchmark results, I matched 611 comparisons of 180 models on 103 benchmarks. AI labs reported scores 6.4% higher on average for the same models on the same tests.

Everything is open: the dataset is CC BY-SA 4.0 and the code is MIT. Sources and their licenses are recorded, with links on each model's page.

Interactive chart: [https://whatmodel.sh/claims](https://whatmodel.sh/claims)

Data and code: [https://github.com/jakemanger/whatmodel](https://github.com/jakemanger/whatmodel)

Please star on Github if you find it useful. Contributions are very much welcome!
```



document fetched 2026-10-20 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x9skrx/ai_labs_report_benchmark_scores_64_higher_than
