← sources

llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch

source: reddit (reddit:LocalLLaMA)published 2026-08-07 17:09ingested 2026-08-08 06:58

retained text

<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1vi6hmw/llamacpp_pr_reports_up_to_169_faster_quantizedkv/"> <img alt="llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch" src="https://preview.redd.it/7xawgw0cjzhh1.jpeg?width=640&amp;crop=smart&amp;auto=webp&amp;s=dbf8a9833542b0efd6b955246a93d173f730d3fd" title="llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch" /> </a> </td><td> <!-- SC_OFF --><div class="md"><p>A fresh llama.cpp PR (#26689) changes what looks like a tiny SYCL FlashAttention dispatch decision.</p> <p>With a quantized KV cache (&quot;q4_0&quot; / &quot;q8_0&quot;), decode was being sent through the VEC kernel. On the author's Battlemage test system, switching that path to TILE gets much faster as context grows.</p> <p>Some of the author-reported results, MTP off:</p> <p>- Qwen3.6-35B, q4_0 KV @ 118,784: 12.99 → 29.61 t/s (+127.9%)</p> <p>- Qwen3.6-35B, q8_0 KV @ 118,784: 12.90 → 31.80 t/s (+146.5%)</p> <p>- Gemma 4 12B, q4_0 KV @ 118,784: 5.06 → 13.59 t/s (+168.7%)</p> <p>- Gemma 4 12B, q8_0 KV @ 118,784: 5.13 → 13.81 t/s (+168.7%)</p> <p>It isn't only the extreme 118K point either. At 32K, the same JIT tests show roughly +42% to +74% on the tested Qwen/Gemma configs.</p> <p>The interesting part is how small the actual idea is. The PR basically changes the dispatch gate so quantized-KV decode selects TILE instead of being forced through VEC, and adds &quot;GGML_SYCL_FA_DECODE_KERNEL=vec|tile|auto&quot; so it can be A/B tested.</p> <p>Big caveats:</p> <p>- PR is open, not merged</p> <p>- these are mostly author-reported benchmarks</p> <p>- the exact Battlemage GPU SKU isn't specified in the PR</p> <p>- this specifically targets quantized KV; F16 keeps the existing dispatch</p> <p>- one 118K MTP test only improved 17.65 → 20.14 t/s (+14.1%)</p> <p>- backend tests pass 4001/4001, but there isn't an independent hardware sweep yet</p> <p>The PR also relays a Laguna-S-2.1 Discord test showing +50% at 64K and +68% at 118K, but I'd still like to see proper independent results.</p> <p>Anyone with a B580 or B70 able to reproduce this at 64K/118K? I'm especially curious whether the huge gain survives with MTP enabled.</p> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/BTA_Labs"> /u/BTA_Labs </a> <br /> <span><a href="https://i.redd.it/7xawgw0cjzhh1.jpeg">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1vi6hmw/llamacpp_pr_reports_up_to_169_faster_quantizedkv/">[comments]</a></span> </td></tr></table>

document fetched 2026-08-08 06:58 · canonical https://reddit.com/r/LocalLLaMA/comments/1vi6hmw/llamacpp_pr_reports_up_to_169_faster_quantizedkv