Llama.cpp setup Qwen-flash-next Vulkan Q4 with Expert cache.
retained text
I just setup my server: 64GB DDR4 2133 4CH (Broadwell Xeon) 2 x RX 6800 16GB 1 x Rx 580 2048sp (16GB) I am using the Q4 of Qwen Flash Next, with the prompt: sudo ./llama-server -m ./models/Qwnflash180b-Q4.gguf --host [0.0.0.0](http://0.0.0.0) \--port 8081 -ngl 99 -fa on --moe-cache-mib 16384 --log-verbosity 4 --metrics -mg 1 I am able to get 19-21t/s generation at full context (~~262K~~ 64K) (which is why i capped the moe-cache-mib at 16GB) **However note: when I set any context size with the -c or --ctx-size flags,** **~~performance tanked to 8t/s even when the context was smaller (100K).~~** **it seems you need to let llama.cpp allocate the context itself.** ***The context was automatically 64k, I was actually increasing it, causing layers to spill over to system ram.*** PCIe slots: gpu0 rx6800 x16 3.0 gpu1 rx6800 x16 3.0 gpu2 rx580 x4 3.0
document fetched 2026-11-02 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1xkr0e6/llamacpp_setup_qwenflashnext_vulkan_q4_with