question: 262K context: the only route that fits uses a 4-bit KV cache, which costs +0.0011 KLD. Is 262K vs 170K context worth the loss?
retained text
I'm building my current engine for my own hardware. The question I want to see how many have answered themselves. Would you take lower (better) KLD on 170K context for Qwen3.8-27B or take 4bit KV cache with .0011 loss on KLD for the 262K context window? Model/Quant: Qwen3.8-27B UD-Q4\_K\_XL v3.1
document fetched 2026-11-19 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xzr9ze/question_262k_context_the_only_route_that_fits