← sources

First local-LLM tuning attempt: Qwen3.8-27B true Q4_K_M at 13.2 tok/s near 50-61K context on RTX 5080 16GB

source: reddit (reddit:LocalLLaMA)published 2026-09-02 14:29ingested 2026-09-03 02:43

retained text

This was my first serious attempt at tuning a local LLM. I started because Qwen3.8-27B IQ3 was fast on my RTX 5080 but the coding quality disappointed me, and the Q4 profiles I tried in LM Studio were much slower than reports here.

Hardware:

- RTX 5080 16 GB
- i5-14600K
- 64 GB DDR5-5600 (4 DIMMs)
- Windows

Final model/runtime:

- Unsloth Qwen3.8-27B UD-Q4_K_M, unmodified (16.46 GB)
- official llama.cpp b10760 CUDA 13.3 build
- 65,536 context, one slot
- Q4_0 K/V cache, Flash Attention
- medium thinking, text only
- Pi as the coding agent

Results:

- 49,738 input tokens: 13.247 / 13.260 / 13.261 tok/s across three runs
- 61,238 input tokens: 13.055 tok/s
- 4/4 retrieval in every run
- Pi read a broken implementation plus a separate test, edited only the implementation, ran PowerShell, and got PASS

The useful change was selective FFN placement. I kept attention/KV and most tensors on the GPU, but moved the 16 largest FFN tensor groups (about 2.764 GiB) to CPU. Whole-layer offload in LM Studio gave me only 6.633 tok/s around 50K.

MTP was surprisingly worse on this machine at deep context. MTP1 reached 8.654 tok/s and MTP3 7.810 tok/s, while disabling MTP reached 13.256 tok/s. My guess is that the CPU-side draft competed for RAM bandwidth with the spilled FFNs.

I originally chased the recent ~75 tok/s 5080 post, but the linked 13.5 GB custom quant uses IQ3_S for its FFN tensors. That is a valid speed tradeoff, but I specifically wanted true Q4 weights and a deep-context measurement.

I published the exact Windows launcher, tensor override, Pi config, benchmark harness, raw results, model SHA, failed profiles, and methodology here:

https://github.com/johnconnor2020/qwen38-27b-rtx5080-16gb

Caveats: the recall prompt is synthetic, the Pi task is a practical smoke test rather than LiveCodeBench/SWE-bench, and runs 2/3 reused prompt cache for ingestion (decode speed stayed the same). This is also likely sensitive to RAM bandwidth and llama.cpp version.

I would be very interested in comparable true-Q4 50K+ results from other 16 GB cards, or suggestions for a better coding-quality benchmark that is practical to run locally.

document fetched 2026-09-03 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1w5c04h/first_localllm_tuning_attempt_qwen3827b_true_q4_k