# A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/ed9bf03697165f23

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-08-07 12:27 ingested 2026-08-08 06:58

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1vhz989/a_llamacpp_pr_makes_q2_0_3036x_faster_on_x86_cpus/)

## retained text



```
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1vhz989/a_llamacpp_pr_makes_q2_0_3036x_faster_on_x86_cpus/"> <img alt="A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s" src="https://preview.redd.it/pyim0m155yhh1.jpeg?width=640&amp;crop=smart&amp;auto=webp&amp;s=0f3388782a63b5b24d4b4732a63dca51dbc4521b" title="A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s" /> </a> </td><td> <!-- SC_OFF --><div class="md"><p>I was going through the current llama.cpp CPU PRs and #26348 stood out because this isn't the usual +5% kernel optimization.</p> <p>It adds an x86 VNNI implementation for the Q2_0 × Q8_0 dot product, and the author's controlled CPU-only benchmarks show roughly 3–3.6x higher throughput across Bonsai models from 1.7B to 27B.</p> <p>Setup:</p> <p>- AMD EPYC 9645</p> <p>- 8 CPU cores</p> <p>- CPU only</p> <p>- GGML_NATIVE=ON</p> <p>- OpenMP enabled</p> <p>- BLAS disabled</p> <p>- -t 8 -ngl 0 -fa off</p> <p>- 3 runs after warmup</p> <p>- group-64 Q2_0 Bonsai GGUFs</p> <p>Results:</p> <p>1.7B</p> <p>pp512: 14.07 → 50.47 tok/s (3.59x)</p> <p>tg128: 10.22 → 33.28 tok/s (3.26x)</p> <p>4B</p> <p>pp512: 5.41 → 19.40 tok/s (3.59x)</p> <p>tg128: 4.45 → 13.36 tok/s (3.00x)</p> <p>8B</p> <p>pp512: 2.82 → 10.26 tok/s (3.64x)</p> <p>tg128: 2.39 → 8.20 tok/s (3.43x)</p> <p>27B</p> <p>pp128: 0.79 → 2.85 tok/s (3.59x)</p> <p>tg32: 0.72 → 2.37 tok/s (3.32x)</p> <p>The 27B baseline was apparently so slow that a pp128 pass took almost 3 minutes.</p> <p>What's actually changing is pretty small: the existing Q2_0 dot product gets a path using AVX-VNNI / AVX-512 VNNI instead of relying on the generic implementation.</p> <p>The reference Prism implementation this was adapted from also exposed an interesting issue on normal consumer Intel CPUs.</p> <p>On an i5-13400, Q2_0 was silently missing the fast path because 12th–14th gen Intel has AVX-VNNI but AVX-512 is fused off. Nothing tells the user this happened, it just looks like Q2_0 is extremely slow.</p> <p>Their controlled i5-13400 A/B:</p> <p>Ternary-Bonsai-8B Q2_0</p> <p>decode: 2.17 → 6.92 tok/s</p> <p>prompt eval: 2.7 → 8.6 tok/s</p> <p>Again, around 3.2x from using the VNNI path.</p> <p>There are some important caveats:</p> <p>- The upstream llama.cpp PR is still open, not merged</p> <p>- This is specifically about Q2_0, not a free 3x for Q4/Q5/etc</p> <p>- The main upstream benchmarks were on an EPYC using only 8 cores</p> <p>- The i5-13400 result comes from the Prism reference implementation, not the exact group-64 upstream PR</p> <p>- There is a tiny numerical difference from fused multiply-add behavior</p> <p>On correctness, the author reports 14,000 randomized comparisons matching bit-for-bit at the kernel level. In the perplexity smoke test the two versions chose the same top token 99.216% +- 0.554% of the time, with a very small KLD difference.</p> <p>This is the kind of llama.cpp optimization I'd really like to see tested on boring consumer hardware rather than another server CPU.</p> <p>If anyone has Alder/Raptor Lake or Zen 4/5 and can compile the PR branch, post your before/after llama-bench results. Especially interested in laptops, does the 3x survive power/memory-bandwidth limits, or shrink a lot in real hardware?</p> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/BTA_Labs"> /u/BTA_Labs </a> <br /> <span><a href="https://i.redd.it/pyim0m155yhh1.jpeg">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1vhz989/a_llamacpp_pr_makes_q2_0_3036x_faster_on_x86_cpus/">[comments]</a></span> </td></tr></table>
```



document fetched 2026-08-08 06:58 · canonical https://reddit.com/r/LocalLLaMA/comments/1vhz989/a_llamacpp_pr_makes_q2_0_3036x_faster_on_x86_cpus
