← sources

CPU-only boost for Q2_0 on AVX-VNNI · Pull Request #26348 · ggml-org/llama.cpp

source: reddit (reddit:LocalLLaMA)published 2026-11-15 16:55ingested 2026-11-16 02:38

retained text

Measurements: mean ± sample standard deviation in tok/s, 3 runs after normal warmup

(16GB of RAM on the AMD EPYC 9645 using eight CPU cores)

|Model|Test|Upstream master|VNNI + VBMI|Speedup|
|:-|:-|:-|:-|:-|
|Bonsai 1.7B|pp512|142.517 ± 0.589|186.373 ± 1.582|1.31×|
|Bonsai 1.7B|tg128|70.435 ± 2.061|68.422 ± 9.055|0.97×|
|Bonsai 4B|pp512|53.430 ± 2.266|71.247 ± 1.925|1.33×|
|Bonsai 4B|tg128|31.655 ± 2.652|38.523 ± 3.820|1.22×|
|Bonsai 8B|pp512|30.010 ± 0.512|39.396 ± 0.424|1.31×|
|Bonsai 8B|tg128|20.630 ± 0.302|23.770 ± 0.209|1.15×|
|Bonsai 27B|pp128|8.502 ± 0.042|11.029 ± 0.073|1.30×|
|Bonsai 27B|tg32|5.340 ± 0.117|5.812 ± 0.324|1.09×|

But noticed few surprising things there. I assumed the boost is only for Ternary models, but it's not. 

>[This makes CPU kernels a lot faster, raising **IQ1\_S/IQ2\_XXS** **DeepSeek v4.1** decode speed](https://github.com/ggml-org/llama.cpp/pull/26348#issuecomment-6300835738)

>[End to end, a 66 GB Q2\_0 MoE (**Qwen3.8-Flash-Next GSQ-RCO Q2\_0**) with --cpu-moe, so all routed experts run through this kernel on the CPU. RTX 4070 Laptop for the rest, 10 threads, MTP on, generation t/s at 2k–8k tokens of real context:](https://github.com/ggml-org/llama.cpp/pull/26348#issuecomment-6415227904)

|CPU path for Q2\_0|TG t/s|
|:-|:-|
|generic scalar (a prebuilt from before [\#30447](https://github.com/ggml-org/llama.cpp/pull/30447))|3.1–4.1|
|AVX2 from [\#30447](https://github.com/ggml-org/llama.cpp/pull/30447) (prebuilt from after it)|**14.0–18.7**|
|an earlier revision of this PR's AVX-VNNI kernel (on an older base)|15.5–21.4|

So check your t/s again if you're using Q1/Q2 quants of big/large models. There would be boost on Hybrid inference(GPU+CPU) too.



AND there's an Open PR related to this for AVX-512 VNNI. Additional boost on the way.

[ggml-cpu: 512-bit AVX-512 VNNI dot for Q2\_0- #32307](https://github.com/ggml-org/llama.cpp/pull/32307)

document fetched 2026-11-16 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1xx0fqn/cpuonly_boost_for_q2_0_on_avxvnni_pull_request