MoE offload on a 3090: prefill 345 -> 466 t/s by tuning cudaMemcpy (Qwen3.6 35B-A3B)
retained text
https://i.redd.it/5dlnj6wict1i1.gif
I made a tool that tunes copies that speeds up CPU offloading.
1 3090 + 7800X3D
llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3
cmt tune -- llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3
before: prefill 345 t/s
after: prefill 466 t/s
blog: https://maknee.github.io/blog/2026/CudaMemcpy/
code: https://github.com/Maknee/cudamemcpy-tunerdocument fetched 2026-11-16 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1xx1owx/moe_offload_on_a_3090_prefill_345_466_ts_by