# New sparse attention method for ultra-long context — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/58c7c340cce22474

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-02 22:24 ingested 2026-11-03 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xmbj0v/new_sparse_attention_method_for_ultralong_context/)

## retained text



```
I am on the team behind the attached open-source research on novel approaches to attention - specifically we show: 

• You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy 

• Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is still fine (interesting insights) 

• You can natively train sparse attention on next-token prediction loss alone and achieve the same validation loss as full attention - but faster

Hope the community likes it - while we focus on H200/B300 GPUs and larger scale system - I wanted to share our research here as well - as it can be applied to small open-weight models (making longer context cheaper on local hardware)

Everything is fully open: kernels / SGLang integration / paper / details 

Hope you find it interesting / useful!

(No hype - we try to be as transparent as possible)
```



document fetched 2026-11-03 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xmbj0v/new_sparse_attention_method_for_ultralong_context
