Improve Qwen3.8-27b low quants by using a prefiller model that streams from SSD
retained text
This demo from ISTA-DASLab recovers low quant accuracy while retaining low VRAM usage and fast decoding. It needs a Nvidia Blackwell GPU to run, so I have not tested it. For context longer than \~4k there should be no speed penalty compare to running without the prefiller model. Graph shows MMLU-Pro accuracy. Huggingface🤗: [https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller)
document fetched 2026-10-21 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xao7kx/improve_qwen3827b_low_quants_by_using_a_prefiller