# What benchmarks do you run locally? — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/77713cecbebc0b63

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-09-16 16:38 ingested 2026-09-17 02:39

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1wi2en4/what_benchmarks_do_you_run_locally/)

## retained text



```
I am trying to get my setup dialed in and want to actually measure some things like quality loss from KV cache quantization (just from BF16 to Q8, I'm not a monster!), performance using different MTP / drafters for prediction, tool calling--generally and with different chat templates (I may still have PTSD from the early-days Qwen 3.6 tool calling issues...), etc. 

I've been especially focused on deterministic benchmarks, as I don't want to be worrying about extra error introduced with LLM-based evals, as well as benchmarks that test performance / quality with large contexts (it seems like a lot limit context or, at least, aren't meant to measure across different context lengths). Over time I will be more thoughtful about turning things I do regularly into something repeatable and quantifiable for testing, but I'm focused on coding and agent capability right now.

So far I have MMLU-Pro, GPQA-Diamond, BCFL v4, and RULER implemented (the first two were more "proof of concept" for adding benchmarks) and plans to add LiveCodeBench, Terminal-Bench, and PlanBench. (And obviously this is all being planned and implemented by my clanker.) But, as the list grows this seems like it could be (a) overkill / over-engineered and (b) could simply be the wrong tests and fail to capture any meaningful difference other than overall model quality versus actually being useful to measure actual difference for different vLLM or llama.cpp configs.

So, I figured I'd ask on here and get the wisdom of everyone who knows way more about these things than I... What do you run? What is reasonable to run locally? What will actually be useful instead of simply being "more data for data's sake"? 
```



document fetched 2026-09-17 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1wi2en4/what_benchmarks_do_you_run_locally
