← sources

How do you locally benchmark and compare local LLMs?

source: reddit (reddit:LocalLLaMA)published 2026-11-08 22:49ingested 2026-11-09 02:37

retained text

If I wanted to locally compare performance for multiple different offline LLM models across various modalities (speed, efficiency, reasoning, tool use, programing/coding, etc), what strategies, prompts, software are commonly used to do this? There are so many models to choose from and run, I'm trying to determine things like whether slower but more complex models are worth running over slightly faster ones, how models that run at similar speed compare to each other's capabilities, and whether any of the custom engines that have been popping up other than llama.cpp/strata are worth it. 

Does this sort of thing exist as something that can be ran locally, and if so, how?

document fetched 2026-11-09 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1xrbhdf/how_do_you_locally_benchmark_and_compare_local