← sources

110+ TPS through 100K tokens with 27b on a single 3090, with 400k ctx

source: reddit (reddit:LocalLLaMA)published 2026-10-28 14:33ingested 2026-10-29 02:42

retained text

Hello everyone! As always, thank you for sharing you previous results (and bugs) on the last version. I am happy to report that LlamAmpere v0.6 is now out.

The headline items for you (at temperature 1):

* **a \~13% jump in TPS, with a 112 TPS** average across 3 task types in the 100-105K token interval
* **a +20% increase on prompt processing**, now starting around 1800 TPS and holding 1000+ through the first 100k
   * The average prefill over the first 100K went from \~1,080 to \~1,300 TPS
* **A new file format (Ampere Integer+) designed to be as fast, smaller, and higher precision (lower KLD) than XS formats**. Vs EXL3, it is approximately the same KLD, with a significantly faster speed.
   * The recommended model (now available on HF) is about 10% better KLD and 1.2 GB smaller. This results in slighter faster speed, and of course more room for context. I will be releasing more quants at smaller sizes in this format later this week as I continue to tweak the design at 2 and 3 bits.
   * I have also upgraded EXL3's kernels, and in my testing it was running \~8% faster than in the original EXL fork. But, at 4 bits this configuration shows \~17% faster decode and 25% faster prefill on AI+, so I will be focused more on those kernels moving forward (EXL had a non-stat sign <2% lower KLD in testing)

As in the last release, these numbers are built around SJKVaRN, which enables a maximum total context of over 400K ctx with 23.5GB total usage (across multiple predictions or context scaling).

I will note that these prefill numbers were performed with the recently create 3090/ti GPU unlock; if you haven't implemented it, I recommend doing so (repo here: [https://github.com/Zagitta/gpu-unlock](https://github.com/Zagitta/gpu-unlock))

Github for LlamAmpere here: [https://github.com/JakeATX/llamAmpere](https://github.com/JakeATX/llamAmpere)

new model here: [https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI4](https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI4)

There are also upgrades relevant for 3.8 FlashNext included, but at present testing is "only" even with the other best options like Strata. I have a couple pending commits that I hope to have out by the end of the day, that should materially boost speed for MOEs models further; I will publish results on that separately.

And for 12 GB cards, I will have some new quants (and possibly a distillation) for 3.8, hopefully by EOW.

Please share your results, and feedback, as usual. Enjoy!

https://preview.redd.it/i1zngu27y7yh1.png?width=1017&format=png&auto=webp&s=fb0275d70ad81f6ee29db2131f1660d85ffbf624

document fetched 2026-10-29 00:44 · canonical https://reddit.com/r/LocalLLaMA/comments/1xhpqhz/110_tps_through_100k_tokens_with_27b_on_a_single