← sources

Qwen Flash Q4_K_M on 4080 + 64GB DDR5 at ~8tk/s 98304 CTX.

source: reddit (reddit:LocalLLaMA)published 2026-08-30 17:31ingested 2026-08-31 02:38

retained text

Managed to cram a 182B model into my small setup. The trick is to offload ngrams into SSD which allows to fit the rest of the model. Believe it or not, this is faster and more intelligent than Qwen3.8 27B. This is great for small GPU users because usually we are locked out of 27B. Note before running this command, make sure to close all unnecessary background tasks, apps and etc.

This is the launch command I used

    llama.exe serve `
      -hf "AtomicChat/Qwen3.8-Flash-Next-GGUF" `
      -hff "Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf" `
      --no-mmproj `
      --offline `
      --load-mode mmap `
      --tensor-read-lazy on `
      --fit off `
      -ngl all `
      -ncmoe 41 `
      -t 16 `
      -tb 16 `
      -c 98304 `
      -b 256 `
      -ub 128 `
      -fa on `
      --jinja `
      --parallel 1 `
      --temp 1 `
      --top-p 0.95 `
      --top-k 20 `
      --min-p 0 `
      --cors-origins localhost `
      --host 127.0.0.1 `
      --port 8080

document fetched 2026-08-31 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1w2nr6e/qwen_flash_q4_k_m_on_4080_64gb_ddr5_at_8tks_98304