Do I need to switch to SGlang/vLLM for Qwen3.8-Flash-Next?
retained text
Setup: 2x5080 + 128gb DDR5
I thought I was cool because I do everything on llama.cpp, but I'm starting to think maybe I should explore vLLM/Sglang seeing how there are not many NVFP4/FP8 .gguf files, and maybe I could be getting better performances out of this model given my setup?
I'm currently getting around 100tps (up to 150-170 if measured as instantaneous rates at different depths) prefill and \~18-20tps decode on Q5\_K\_XL using MTP and ncmoe42 on llama.cpp:
\[Qwen3.8-Flash-Next-UD-Q5\_K\_XL-dual-mtp-ncmoe42-vision\]
alias = Qwen3.8-Flash-Next-Q5-dual-mtp-vision
model = models/mymodels/unsloth/Qwen3.8-Flash-Next-UD-Q5\_K\_XL/Qwen3.8-Flash-Next-UD-Q5\_K\_XL-00001-of-00006.gguf
load-mode = mmap
mlock = false
no-mmap = false
lazy-mode = on
fit = off
ngl = 999
n-cpu-moe = 42
override-tensor = per\_layer\_token\_embd=CPU,blk\\.(42|43|44|45)\\.ffn\_(up|down|gate)\_exps=CUDA0,blk\\.(46|47)\\.ffn\_(up|down|gate)\_exps=CUDA1
device = CUDA0,CUDA1
tensor-split = 1,1
ctx-size = 262144
batch-size = 2048
ubatch-size = 512
cache-type-k = q8\_0
cache-type-v = q8\_0
flash-attn = on
threads = 8
parallel = 1
mmproj = models/mymodels/unsloth/mmproj-Qwen3.8-Flash-Next-BF16.gguf
spec-type = draft-mtp
spec-draft-model = models/mymodels/unsloth/Qwen3.8-Flash-Next-MTP/mtp-Qwen3.8-Flash-Next-shared-Q4\_K\_M.gguf
spec-draft-n-max = 2
jinja = true
reasoning = on
chat-template-kwargs = {"enable\_thinking": true, "preserve\_thinking": true, "reasoning\_effort": "medium"}
temp = 1.0
top-k = 20
top-p = 0.95
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0document fetched 2026-09-18 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1wiu615/do_i_need_to_switch_to_sglangvllm_for