← sources

CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows by anujj · Pull Request #29948 · ggml-org/llama.cpp

source: reddit (reddit:LocalLLaMA)published 2026-10-25 09:35ingested 2026-10-26 02:40

retained text

# 

time to download some speedup

# 

DGX Spark (GB10), `llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5`, tok/s:

|model|type|master|PR||
|:-|:-|:-|:-|:-|
|Qwen3.8-27B|Q4\_K\_M|840.1|899.9|\+7.1%|
||Q5\_K\_M|840.1|880.6|\+4.8%|
||Q6\_K|751.2|785.8|\+4.6%|
||Q8\_0|876.2|940.1|\+7.3%|
||IQ4\_XS|882.4|948.0|\+7.4%|
|Qwen3.6-27B|Q4\_K\_M|816.6|865.6|\+6.0%|
||Q5\_K\_M|800.0|851.1|\+6.4%|
||Q6\_K|720.8|770.4|\+6.9%|
||Q8\_0|876.9|940.2|\+7.2%|
||IQ4\_XS|882.2|948.5|\+7.5%|
|Llama-3.1-8B|Q4\_K\_M|2749.3|3000.6|\+9.1%|
||Q5\_K\_M|2699.4|2960.3|\+9.7%|
||Q6\_K|2392.2|2616.0|\+9.4%|
||IQ4\_XS|2933.9|3249.0|\+10.7%|

RTX 5090, `llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5`, tok/s:

|model|type|master|PR||
|:-|:-|:-|:-|:-|
|Qwen3.8-27B|Q4\_K\_M|3688.4|3790.9|\+2.8%|
||Q5\_K\_M|3589.5|3659.6|\+2.0%|
||Q6\_K|3220.5|3281.0|\+1.9%|
||Q8\_0|3617.3|3766.9|\+4.1%|
||IQ4\_XS|3796.2|3926.8|\+3.4%|
|Qwen3.6-27B|Q4\_K\_M|3567.9|3660.3|\+2.6%|
||Q5\_K\_M|3429.8|3525.3|\+2.8%|
||Q6\_K|3105.2|3187.8|\+2.7%|
||Q8\_0|3628.7|3788.4|\+4.4%|
||IQ4\_XS|3785.6|3915.4|\+3.4%|
|Llama-3.1-8B|Q4\_K\_M|12071.5|12625.9|\+4.6%|
||Q5\_K\_M|11823.0|12297.3|\+4.0%|
||Q6\_K|10513.1|10928.6|\+4.0%|
||IQ4\_XS|12997.6|13603.3|\+4.7%|

document fetched 2026-10-26 00:38 · canonical https://reddit.com/r/LocalLLaMA/comments/1xey0uw/cuda_fuse_ffn_gateup_matmuls_and_glu_in_mmq_with