← sources

I ran all the popular Qwen 3.8 FN engines on a single 5090 + 128GB of RAM, here are the numbers

source: reddit (reddit:LocalLLaMA)published 2026-10-23 18:22ingested 2026-10-24 02:40

retained text

>**Before you read:** I'm the developer of Basalt, one of the engines benchmarked below. I tried to keep this as fair as possible: same checkpoint, GPU and requests, and every engine set up the way its own docs recommend. How it was run is at the bottom. This post has had some AI cleanup from my own wording.

I wanted to get a comparison on all of the Qwen 3.8 FN engines I hear people talking about, so after 5 or so excruciating hours of setup (remind me not to do this ever again), I benchmarked all of the ones I know of: Strata, Infernix, capy-flash, ninfer-ext, ninfer, llama.cpp and Basalt. Each one converted NVIDIA's NVFP4 with its own tool, then got the same seven greedy requests on an RTX 5090 at 400 W, one cold start per request. Anyway, here are the results:

# Decode, short context

[Decode, short context](https://preview.redd.it/21o6k7qeb9xh1.png?width=2400&format=png&auto=webp&s=edbee2f1ff016d726cd863da8ef2a83794178e8e)

# Decode at context depth

[Decode at context depth](https://preview.redd.it/vb50mligb9xh1.png?width=2400&format=png&auto=webp&s=58b12dfd9975ea942f15bf4b9f34ab640e1b0b5d)

# Prefill at context depth

[Prefill at context depth](https://preview.redd.it/e08rg50ib9xh1.png?width=2400&format=png&auto=webp&s=006318a80c6329244aee9c2a6f5b613d831e41ef)

# All the numbers

Best value per column in bold.

**Decode, tokens per second**

|Engine|Struct|Conv|Prose|25K|50K|100K|200K|
|:-|:-|:-|:-|:-|:-|:-|:-|
|Basalt|**354**|**188**|**180**|**190**|**187**|**184**|**160**|
|Infernix|349|170|163|163|174|176|158|
|capy-flash|292|165|152|152|143|152|135|
|Strata|204|146|123|130|123|121|114|
|ninfer-ext|140|101|95|92|86|79|66|
|ninfer|54|77|45|53|46|47|44|
|llama.cpp|39|39|39|37|36|35|32|

**Prefill, tokens per second**

|Engine|25K|50K|100K|200K|
|:-|:-|:-|:-|:-|
|Basalt|6,492|**7,662**|**8,666**|**8,609**|
|Infernix|**6,493**|6,544|6,630|6,436|
|capy-flash|4,026|4,272|5,206|5,458|
|Strata|3,530|3,738|4,129|4,245|
|ninfer-ext|3,552|4,049|4,683|4,588|
|ninfer|2,080|2,236|2,308|2,248|
|llama.cpp|1,354|1,420|1,394|1,257|

**Time to first token, seconds**

|Engine|Struct|Conv|Prose|25K|50K|100K|200K|
|:-|:-|:-|:-|:-|:-|:-|:-|
|Basalt|**0.35**|0.23|0.17|**3.94**|**6.65**|**11.81**|**23.79**|
|Infernix|0.42|0.24|**0.16**|**3.94**|7.79|15.43|31.81|
|capy-flash|0.88|0.50|0.29|6.35|11.93|19.65|37.52|
|Strata|0.45|0.29|0.20|7.24|13.64|24.78|48.24|
|ninfer-ext|1.05|0.47|0.21|7.19|12.59|21.85|44.63|
|ninfer|1.78|**0.19**|0.17|12.28|22.80|44.32|91.07|
|llama.cpp|1.09|0.75|0.31|18.87|35.91|73.39|162.86|

# Takeaways

* Infernix and Basalt are ahead of the rest. Against Infernix, Basalt decodes 10% faster on prose and 10% on conversation, struct is roughly a tie (354 vs 349).
* Prefill at 25K is a tie (3.94 s vs 3.94 s to the first token). From 50K up Basalt pulls away.
* capy-flash was interesting, was the closest of the rest on decode, it is built mostly for two GPUs, so this single GPU run is not its best case.
* llama.cpp ran with MTP off, so it's decode is not comparable with the others. This was just a reference point.

# How it was run

* Model: Qwen3.8 FN, NVIDIA's NVFP4 converted for each engine (table below).
* Hardware: RTX 5090, PL 400 W, 128 GB RAM, Linux.
* Sampling: greedy (temperature 0), thinking off, int8 KV cache for every engine (llama ran f16 kv).
* Cold: a fresh server per request, no warm up, one run each. Decode varies between runs, so any gaps under 5% are ties.
* Depth tests: a wikitext prefix of about 25K / 50K / 100K / 200K tokens, then "Write an essay on the Odyssey.", 512 tokens out.

**What each engine ran**

|Engine|Commit|Weights|Drafter|Note|
|:-|:-|:-|:-|:-|
|Basalt|9c3692e7|NVIDIA NVFP4 experts bit-exact, dense and PLE table 8-bit (Q8\_0)|own MTP drafter (spec 8, shipped default)|Fair pack built with Basalt's converter|
|Infernix|7dbe99a3|Dense8 conversion: experts and FP8 PLE bit-exact, dense 8-bit|own DFlash2 drafter|Its README's best setup|
|capy-flash|4e45286a|Own convert: indexer BF16, GDN 8-bit|own re-distilled MTP head|Single-GPU profile, built mainly for two-GPU tensor parallel|
|Strata|011c9e60|Experts bit-exact, dense Q8\_0, PLE from an IQ4\_NL GGUF|Qwen BF16 MTP head repacked|Basalt's upstream|
|ninfer-ext|bc168661|Own convert: MTP NVFP4, PLE re-scaled FP8, projections 8-bit|own MTP||
|ninfer|64d10fbf|Patched to read NVIDIA's experts and fix the PLE scale|own MTP|Stock dev head did not build, stock converter needs the BF16 base|
|llama.cpp|36cbf415|magnitudedev NVFP4 GGUF: experts bit-exact, dense and PLE Q8\_0|none||

document fetched 2026-10-24 10:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1xdlunp/i_ran_all_the_popular_qwen_38_fn_engines_on_a