How much can I quantise my KV cache before it gets degraded?
retained text
My system can barely handle the model(Qwen3.8 27B GSQ-RCO IQ3\_XXS) as it is. But I also need to increase its context size to 128k to get serious work done. I quantised the KV cache down to 8 bits and it seems to work. Now, I'm thinking if I should quantise down to 4-bit or keep it as it is.
document fetched 2026-09-19 00:40 · canonical https://reddit.com/r/LocalLLaMA/comments/1wjqpad/how_much_can_i_quantise_my_kv_cache_before_it