← sources

Strix Halo int4 cores: Gemma 4 31B at 2.43x llama.cpp, same quality

source: reddit (reddit:LocalLLaMA)published 2026-11-17 10:08ingested 2026-11-18 02:42

retained text

The 8060S has an int4 x int4 WMMA instruction at 2x the FP16 rate (we measured 104 TOPS). Nothing in ROCm or llama.cpp uses it. We wrote a GEMM for it and run Gemma 4 with int4 weights and int4 activations. To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision.

llama-bench, Gemma 4 31B QAT Q4\_0 (llama.cpp b11056 Vulkan, -fa 1). Server benchmark, 2k-token prompts, greedy, single stream:

||pp512|pp2048|
|:-|:-|:-|
|llama.cpp|287|276|
|ours|679|724|

Full test sets, same settings, 31B:

||GSM8K|IFEval|MMLU-Pro|
|:-|:-|:-|:-|
|llama.cpp|96.2|92.2|85.1|
|ours|96.4|91.7|85.1|

Run it:

    curl -fsSL https://embedl.com/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"
    embedl login
    embedl serve google/gemma-4-31B-it/ryzen-ai-max-395 -d
    embedl infer google/gemma-4-31B-it/ryzen-ai-max-395 --prompt "Explain mixture-of-experts in one sentence."
    embedl bench google/gemma-4-31B-it/ryzen-ai-max-395

12B, E4B and E2B are up too (2.80x / 1.58x / 1.39x end to end, in the chart). Swap the name, e.g. `google/gemma-4-12B-it/ryzen-ai-max-395`.

Model, methodology, and more: [https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395](https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395)

document fetched 2026-11-18 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xygpmv/strix_halo_int4_cores_gemma_4_31b_at_243x