← sources

Benchmarks are misleading?

source: reddit (reddit:LocalLLaMA)published 2026-09-16 16:23ingested 2026-09-17 02:39

retained text

I've been playing around with different models and I'm starting to really distrust the benchmarks. For example Qwen 3.8 27b Scores above on benchmarks compared to Muse 30B. But in my experimentation I have found that Muse is exponentially better when it comes to adhering longer context and following multi step reasoning. Even silly things like asking multiple questions in one prompt and you will get \~50% of the time answer to only one of them because Qwen will spend most of its thinking on one of the issues and answer only the hard question, which I'm guessing is a byproduct of benchmarkmaxxing? 

What has your experience been with different models? Which one do you think is a sleeper or something that has gone unnoticed just because of the benchmark scoring hype?

document fetched 2026-09-17 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1wi1zjk/benchmarks_are_misleading