# Running Qwen3.8 27b on 12GB with full context, 70 TPS, MTP, and 50% lower KLD — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/26ec485af4c7797c

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-17 19:20 ingested 2026-11-18 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xytjrk/running_qwen38_27b_on_12gb_with_full_context_70/)

## retained text



```
Hello everyone. LlamAmpere v0.7 is out with a series of updates focus on the small card users: a new model for the 12GB cards that is comfortably best in class for its size, updates to the runtime to support the previous\* 2.3 bpw model (originally designed for 12GB) on 10GB cards. I am working on an 8GB size distillation and hope to have that done in the next couple of days.

The quick rundown for you:

https://preview.redd.it/vk3azmwn382i1.png?width=973&format=png&auto=webp&s=51f80ad2686202558afebee4bdaa352220884a61



**Vs the October 9 2.3 bpw file, which was the best 2.3 file that I tested for 12GB with long context and MTP** (though 15% worse than the \~11% larger Mirai file), **this most recent file for the 12GB segment is nearly 70% lower error** (>50% vs the previous fit \~2 weeks ago), with a top 1 match of \~96% vs BF16 (as before, this quant is based on the Swift distill, which cuts token verbosity 30-50%). This 12GB class model is noticeably nicer to use in harnesses like hermes and pi than any other model I’ve been able to fit previously.

Vs the recently upgraded v3.1 unsloth quants, it is \~25% lower error despite being 8% smaller on card. **This is while supporting \~200k ctx and MTP.** (note that the embedding table is kept in RAM at high fidelity to add quality without a meaningful hit to speed)

There are a number of tricks involved to make this possible, but 2 I will call out in particular: v0.7 uses both dynamic microbatching and Latent Sparse Attention.

The first one is straightforward: the model starts with the largest microbatch (ub) that will fit on the card to maximize initial prefill, and then periodically resets to smaller batches as context fills up. This frees up hundreds of MB for more context at the end without nuking your prefill for the entire chat.

The second is based on the paper by [prefontal](https://github.com/Prefrontal-AI/latent-sparse-attention) AI that was recently published. **LSA moves from full dense attention to sparse attention**; it projects tokens through a tiny learned map (distilled from the model's own attention) into a 64-dim "index space". It then groups past tokens into small clusters, scores the current query against each cluster, and runs normal, exact attention only on the best clusters. **The initial attempts I ran with this showed the same problem many users have noticed with Qwen Flash Next: sparse attention can struggle at deep context.** Unlike other sparse attention implementations, which only use \~2K tokens, **LlamAmpere is currently using the most recent 8K tail, the 128 first initial tokens, and 20% of the intermediate tokens to protect quality**. This showed significantly stronger attention performance (10-40x less error from attention, depending on the test), at the cost of somewhat slower attention. With a cache for recently attended to tokens, **we can push most of the context out of the card** with modest quality and speed penalties. This allows a much more capable model to fit.

As before, the recommended config uses the AmperInteger+ trellis format and SJKVaRN to efficiently compress the weights and KV.

These same techniques yield benefits for the 10GB class, as what was once required 12GB now fits on 10. The previous 2.3bpw quant now fits 213K context (headless) on 10GB (180K headed). 12GB cards can support the full 262K headless and 180k headed. This model previously tested to perform in line [with 3.6 BF16](https://www.reddit.com/r/LocalLLaMA/comments/1xlerw1/an_updated_23bpw_model_for_qwen_27b_12_gb_cards/), so it is a very solid quant.

As before, the code is available on github, with an MIT license.

And of course, as soon as qwen4 27b comes out, I'll get back to work on quantizing them similarly.

[https://github.com/JakeATX/llamAmpere](https://github.com/JakeATX/llamAmpere)

12gb model (3bpw): [https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI3](https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI3)

10GB model (2.3bpw): [https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2](https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2)

Enjoy!
```



document fetched 2026-11-18 00:45 · canonical https://reddit.com/r/LocalLLaMA/comments/1xytjrk/running_qwen38_27b_on_12gb_with_full_context_70
