CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows by anujj · Pull Request #29948 · ggml-org/llama.cpp
retained text
# time to download some speedup # DGX Spark (GB10), `llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5`, tok/s: |model|type|master|PR|| |:-|:-|:-|:-|:-| |Qwen3.8-27B|Q4\_K\_M|840.1|899.9|\+7.1%| ||Q5\_K\_M|840.1|880.6|\+4.8%| ||Q6\_K|751.2|785.8|\+4.6%| ||Q8\_0|876.2|940.1|\+7.3%| ||IQ4\_XS|882.4|948.0|\+7.4%| |Qwen3.6-27B|Q4\_K\_M|816.6|865.6|\+6.0%| ||Q5\_K\_M|800.0|851.1|\+6.4%| ||Q6\_K|720.8|770.4|\+6.9%| ||Q8\_0|876.9|940.2|\+7.2%| ||IQ4\_XS|882.2|948.5|\+7.5%| |Llama-3.1-8B|Q4\_K\_M|2749.3|3000.6|\+9.1%| ||Q5\_K\_M|2699.4|2960.3|\+9.7%| ||Q6\_K|2392.2|2616.0|\+9.4%| ||IQ4\_XS|2933.9|3249.0|\+10.7%| RTX 5090, `llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5`, tok/s: |model|type|master|PR|| |:-|:-|:-|:-|:-| |Qwen3.8-27B|Q4\_K\_M|3688.4|3790.9|\+2.8%| ||Q5\_K\_M|3589.5|3659.6|\+2.0%| ||Q6\_K|3220.5|3281.0|\+1.9%| ||Q8\_0|3617.3|3766.9|\+4.1%| ||IQ4\_XS|3796.2|3926.8|\+3.4%| |Qwen3.6-27B|Q4\_K\_M|3567.9|3660.3|\+2.6%| ||Q5\_K\_M|3429.8|3525.3|\+2.8%| ||Q6\_K|3105.2|3187.8|\+2.7%| ||Q8\_0|3628.7|3788.4|\+4.4%| ||IQ4\_XS|3785.6|3915.4|\+3.4%| |Llama-3.1-8B|Q4\_K\_M|12071.5|12625.9|\+4.6%| ||Q5\_K\_M|11823.0|12297.3|\+4.0%| ||Q6\_K|10513.1|10928.6|\+4.0%| ||IQ4\_XS|12997.6|13603.3|\+4.7%|
document fetched 2026-10-26 00:38 · canonical https://reddit.com/r/LocalLLaMA/comments/1xey0uw/cuda_fuse_ffn_gateup_matmuls_and_glu_in_mmq_with