← sources

Improve Qwen3.8-27b low quants by using a prefiller model that streams from SSD

source: reddit (reddit:LocalLLaMA)published 2026-10-20 10:09ingested 2026-10-21 02:41

retained text

This demo from ISTA-DASLab recovers low quant accuracy while retaining low VRAM usage and fast decoding.

It needs a Nvidia Blackwell GPU to run, so I have not tested it.

For context longer than \~4k there should be no speed penalty compare to running without the prefiller model.

Graph shows MMLU-Pro accuracy.

Huggingface🤗: [https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller)

document fetched 2026-10-21 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xao7kx/improve_qwen3827b_low_quants_by_using_a_prefiller