← sources

Qwen3.8 Flash on 12GB VRAM - 15 tokens/s

source: reddit (reddit:LocalLLaMA)published 2026-09-16 17:41ingested 2026-09-17 02:39

retained text

Achieved steady 15 tokens/second output and 100-120 promp processing per second with 12GB GPU (RTX 5070 SFF) with Qwen3.8-Flash-Next-GSQ-RCO-GGUF at 3 bpw (IQ3\_XXS which maches BF16 on AIME25). Around 76GB full gguf - while only 47GB needs to be sharded (loaded into VRAM+RAM) rest is done in SSD.

Output quality is really good - under my tests it matches Unsloth’s Q6-Q8 level. 

Hardware:

64GB DDR5 (5600)  
12GB - RTX 5070 SFF Gigabyte  
1TB NVME  
Ryzen 5 7600 CPU

Outputs and results:

| **Server startup** | 4.9s |  
| **Cold launch** → 1K prompt | 31.7s** **|  
| **1K generation** | 13.8 tokens/s |  
| **8K generation** | 14.6 tokens/s |  
| **20K prompt processing** | 104.7 tokens/s |  
| **20K wait before output** | 3,11min |  
| **20K generation** | 14.0 tokens/s |  
| **90–100K generation** | 11.3 tokens/s |

128K max context length in my testing.

Use FreeToken CLI with llama.

Testing 2.40bpw version soon which should be equivalent to NVFP4 - Q4 quants and keep you posted.

If anyone runs similar things let me know your experiences.

document fetched 2026-09-17 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1wi46on/qwen38_flash_on_12gb_vram_15_tokenss