← sources

New sparse attention method for ultra-long context

source: reddit (reddit:LocalLLaMA)published 2026-11-02 22:24ingested 2026-11-03 02:42

retained text

I am on the team behind the attached open-source research on novel approaches to attention - specifically we show: 

• You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy 

• Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is still fine (interesting insights) 

• You can natively train sparse attention on next-token prediction loss alone and achieve the same validation loss as full attention - but faster

Hope the community likes it - while we focus on H200/B300 GPUs and larger scale system - I wanted to share our research here as well - as it can be applied to small open-weight models (making longer context cheaper on local hardware)

Everything is fully open: kernels / SGLang integration / paper / details 

Hope you find it interesting / useful!

(No hype - we try to be as transparent as possible)

document fetched 2026-11-03 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xmbj0v/new_sparse_attention_method_for_ultralong_context