Qwen3.8 Flash on 12GB VRAM - 15 tokens/s
retained text
Achieved steady 15 tokens/second output and 100-120 promp processing per second with 12GB GPU (RTX 5070 SFF) with Qwen3.8-Flash-Next-GSQ-RCO-GGUF at 3 bpw (IQ3\_XXS which maches BF16 on AIME25). Around 76GB full gguf - while only 47GB needs to be sharded (loaded into VRAM+RAM) rest is done in SSD. Output quality is really good - under my tests it matches Unsloth’s Q6-Q8 level. Hardware: 64GB DDR5 (5600) 12GB - RTX 5070 SFF Gigabyte 1TB NVME Ryzen 5 7600 CPU Outputs and results: | **Server startup** | 4.9s | | **Cold launch** → 1K prompt | 31.7s** **| | **1K generation** | 13.8 tokens/s | | **8K generation** | 14.6 tokens/s | | **20K prompt processing** | 104.7 tokens/s | | **20K wait before output** | 3,11min | | **20K generation** | 14.0 tokens/s | | **90–100K generation** | 11.3 tokens/s | 128K max context length in my testing. Use FreeToken CLI with llama. Testing 2.40bpw version soon which should be equivalent to NVFP4 - Q4 quants and keep you posted. If anyone runs similar things let me know your experiences.
document fetched 2026-09-17 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1wi46on/qwen38_flash_on_12gb_vram_15_tokenss