← sources

Strix Halo-specific kernels for vLLM: up to 31× faster prefill at 64k, plugs into stock vLLM

source: reddit (reddit:LocalLLaMA)published 2026-10-29 06:59ingested 2026-10-30 02:40

retained text

I've had a Strix Halo box (Radeon 8060S, 128 GB unified) for a while. vLLM runs on it, but slowly, because the stock ROCm kernels are written with MI300 in mind and not this APU. Long-context prefill was the worst of it. A 256k prompt on Qwen3.6-35B-A3B took more than two hours.

So I spent the last few weeks writing HIP kernels just for gfx1151 and wrapped them in a vLLM plugin. It's called **libstrix**.

**How it works**

* You `pip install` it into the same env as your vLLM. There's no fork and you don't patch anything. It hooks in through vLLM's normal plugin system.
* It only turns on with ROCm + gfx1151. On any other GPU it does nothing.
* Each kernel is built for one specific shape and refuses anything else. If a layer doesn't match, vLLM just uses its own kernel. In my testing nothing got slower than stock, single stream or batched.
* It covers MoE GEMMs, attention prefill and decode, the Qwen GDN layers, and the int8 / int4 / MXFP4 linears.
* `STRIX_DISABLE=all` turns it off, so you can A/B it against stock yourself.

**Numbers**

Stock upstream vLLM vs. the same vLLM with the plugin, single stream, one APU. Every cell is **stock → libstrix in tok/s**. "Weights" is the checkpoint's format and "KV" is the KV cache dtype.

|model|weights|KV|prefill @2k|prefill @64k|decode @2k|decode @64k|
|:-|:-|:-|:-|:-|:-|:-|
|Qwen3.6-35B-A3B|bf16|bf16|1,583 → 3,581|260 → 2,617|17.8 → 32.6|12.8 → 26.9|
||bf16|int8|1,331 → 3,589|132 → 2,558|17.6 → 32.8|12.5 → 29.3|
|Qwen3.6-35B-A3B|int8|bf16|1,824 → 3,214|263 → 2,485|29.3 → 29.3|18.2 → 25.0|
||int8|int8|1,504 → 3,221|133 → 2,432|29.0 → 29.7|17.6 → 27.0|
|Qwen3.8-27B|bf16|bf16|514 → 677|55 → 533|4.3 → 4.4|3.5 → 4.0|
||bf16|int8|423 → 709|38 → 539|4.3 → 4.4|3.6 → 4.1|
|Qwen3.8-27B + MTP|int8|bf16|304 → 694|50 → 537|3.3 → 7.7|2.8 → 6.1|
||int8|int8|270 → 692|37 → 532|3.3 → 7.8|2.8 → 6.6|
|Qwen3.8-27B (Quark AWQ)|int4|bf16|380 → 690|47 → 534|12.8 → 12.9|7.6 → 8.8|
||int4|int8|335 → 688|37 → 528|12.8 → 13.0|8.2 → 10.1|
|LFM2.5-24B-A2B|bf16|bf16|3,208 → 5,007|304 → 3,615|18.0 → 38.3|16.3 → 31.5|
||bf16|int8|2,530 → 5,124|311 → 3,540|17.8 → 38.3|14.5 → 31.7|
|gemma-4-26B-A4B|bf16|bf16|1,548 → 3,591|140 → 2,108|22.8 → 24.6|12.1 → 21.2|
||bf16|int8|890 → 3,877|75 → 2,336|20.3 → 25.9|7.1 → 23.5|
|gemma-4-26B-A4B AWQ|int4|bf16|1,356 → 3,395|138 → 2,225|32.6 → 37.2|14.6 → 29.7|
||int4|int8|867 → 3,358|76 → 2,151|28.1 → 38.2|7.8 → 33.3|
|gemma-4-31B QAT|int4|bf16|245 → 542|31 → 389|9.0 → 10.4|5.0 → 8.1|
||int4|int8|179 → 535|17 → 382|8.2 → 10.8|3.1 → 9.2|
|gemma-4-31B|MXFP4|bf16|289 → 492|33 → 397|1.6 → 9.4|1.4 → 7.4|
||MXFP4|int8|203 → 542|19 → 389|1.6 → 9.7|1.2 → 8.3|

Some absolute numbers with the plugin at 128k context and int8 KV:

* Qwen3.6-35B-A3B: 1,751 tok/s prefill, 26 tok/s decode
* LFM2.5-24B-A2B: 2,405 tok/s prefill, 27 tok/s decode
* gemma-4-26B-A4B AWQ: 1,499 tok/s prefill, 30 tok/s decode

At 256k, the 35B now prefills in about 4 minutes (241 s). Stock vLLM took 7,568 s.

**Accuracy**

I measured KLD against stock vLLM. On the Qwen models it's at the noise floor (0.001–0.006, 97–99% top-1 agreement). GSM8K with the plugin on vs. off shows no significant difference on any model I tested. Gemma-4-26B drifts more (KLD around 0.2). Gemma is unusually sensitive to tiny bf16 rounding differences.

**What it doesn't do**

* Decode isn't always faster. On the 27B bf16, the 27B int4 and the 35B int8, decode is basically the same as stock, because none of the kernels I replace are the bottleneck there. The big decode gains show up where they are.
* The MXFP4 decode number is inflated. Stock vLLM runs MXFP4 through a slow emulation path on this GPU, so I'm beating a weak baseline there.
* It's gfx1151 only (the one I tested), not tested on other GPUs.
* You have to build it against the same torch and ROCm that your vLLM uses, or it fails at import.

**Repo:** [https://github.com/asf0/libstrix](https://github.com/asf0/libstrix) (MIT)

document fetched 2026-10-30 00:45 · canonical https://reddit.com/r/LocalLLaMA/comments/1xicvut/strix_halospecific_kernels_for_vllm_up_to_31