Strix Halo int4 cores: Gemma 4 31B at 2.43x llama.cpp, same quality
retained text
The 8060S has an int4 x int4 WMMA instruction at 2x the FP16 rate (we measured 104 TOPS). Nothing in ROCm or llama.cpp uses it. We wrote a GEMM for it and run Gemma 4 with int4 weights and int4 activations. To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision.
llama-bench, Gemma 4 31B QAT Q4\_0 (llama.cpp b11056 Vulkan, -fa 1). Server benchmark, 2k-token prompts, greedy, single stream:
||pp512|pp2048|
|:-|:-|:-|
|llama.cpp|287|276|
|ours|679|724|
Full test sets, same settings, 31B:
||GSM8K|IFEval|MMLU-Pro|
|:-|:-|:-|:-|
|llama.cpp|96.2|92.2|85.1|
|ours|96.4|91.7|85.1|
Run it:
curl -fsSL https://embedl.com/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"
embedl login
embedl serve google/gemma-4-31B-it/ryzen-ai-max-395 -d
embedl infer google/gemma-4-31B-it/ryzen-ai-max-395 --prompt "Explain mixture-of-experts in one sentence."
embedl bench google/gemma-4-31B-it/ryzen-ai-max-395
12B, E4B and E2B are up too (2.80x / 1.58x / 1.39x end to end, in the chart). Swap the name, e.g. `google/gemma-4-12B-it/ryzen-ai-max-395`.
Model, methodology, and more: [https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395](https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395)document fetched 2026-11-18 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xygpmv/strix_halo_int4_cores_gemma_4_31b_at_243x