# Qwen3.8-FN at NVFP4 ( Basalt vs cachy-flash vs ninfer-ext) — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/860c6373d273f30d

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-22 17:29 ingested 2026-10-23 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xcp0mf/qwen38fn_at_nvfp4_basalt_vs_cachyflash_vs/)

## retained text



```
# Qwen3.8-Flash-Next NVFP4 on one RTX 5090 - prefill and decode against context

Full 262,144-token context, one request at a time, MTP speculation on, int8 KV, identical test for all three engines.

Every number is tokens/second. Prefill = prompt tokens / time to first token. Decode = generated tokens / decode time.

The fastest engine in each row is in bold.

Each data point is a fresh, non-repeating prompt: the servers reported zero prefix-cache hits.

| Item | Value |
|---|---|
| GPU | NVIDIA RTX 5090, 32 GB |
| CPU, RAM | Intel Core Ultra 9 285K, 247 GB DDR5 |
| Model | Qwen3.8-Flash-Next, NVIDIA NVFP4 experts, 48 layers, 512 experts |
| Artifacts | basalt `Qwen3.8-FN-NVFP4.basalt`, capy-flash `qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer`, ninfer-ext `qwen3_8_flash_next_nvfp4_draft.ninfer` |
| Context | 262,144 tokens, swept in 16k steps (16 points) |
| Output | 256 tokens per request, ignore_eos |
| Builds | basalt `fe7fd96f`, capy-flash `4e45286a` v0.7.1, ninfer-ext `3241c511` |

## Engine configuration (after tuning)

| Setting | basalt | capy-flash | ninfer-ext |
|---|---|---|---|
| max context | 262,144 | 262,144 | 262,144 |
| KV cache | int8, elastic | int8, elastic | int8 |
| expert cache | auto | auto (elastic) | auto |
| speculation | MTP, spec window 7 | MTP, 3 drafts | MTP, 3 drafts |
| CPU side | pool-workers 15, ple-io ram | miss-split auto, 16 CPU threads | ngram mapped |
| VRAM used | 27.1 GiB + 40.7 GiB host RAM | 28.3 GiB + 65.2 GiB host | 23.1 GiB |

## Greedy decoding, temperature 0

### Prefill, tokens/s

| Context (tokens) | capy-flash | basalt | ninfer-ext |
|---|---|---|---|
| 16k | 6,921 | **8,490** | 4,473 |
| 32k | 7,736 | **8,712** | 5,193 |
| 48k | 8,015 | **9,211** | 5,764 |
| 64k | 7,976 | **9,052** | 6,113 |
| 80k | 7,806 | **9,206** | 6,269 |
| 96k | 7,879 | **8,949** | 5,842 |
| 112k | 7,800 | **9,061** | 5,353 |
| 128k | 7,677 | **8,978** | 5,696 |
| 144k | 7,238 | **8,842** | 5,321 |
| 160k | 7,429 | **8,773** | 5,311 |
| 176k | 7,391 | **8,670** | 5,182 |
| 191k | 7,283 | **8,553** | 5,187 |
| 207k | 7,207 | **8,511** | 5,057 |
| 224k | 7,063 | **8,434** | 4,847 |
| 239k | 6,912 | **8,347** | 4,846 |
| 255k | 6,828 | **8,285** | 4,748 |

### Decode, tokens/s

| Context (tokens) | capy-flash | basalt | ninfer-ext |
|---|---|---|---|
| 16k | 198 | **242** | 92 |
| 32k | 186 | **265** | 96 |
| 48k | 170 | **251** | 106 |
| 64k | 160 | **246** | 99 |
| 80k | 167 | **242** | 80 |
| 96k | 157 | **205** | 87 |
| 112k | 157 | **225** | 90 |
| 128k | 165 | **216** | 88 |
| 144k | 157 | **202** | 88 |
| 160k | 151 | **187** | 87 |
| 176k | 151 | **199** | 78 |
| 191k | 162 | **196** | 83 |
| 207k | 152 | **185** | 76 |
| 224k | 151 | **187** | 74 |
| 239k | 144 | **177** | 73 |
| 255k | 138 | **176** | 75 |

## Sampled decoding, temperature 1.0, top_p 0.95, top_k 20

### Prefill, tokens/s

| Context (tokens) | capy-flash | basalt | ninfer-ext |
|---|---|---|---|
| 16k | 6,881 | **8,571** | 4,198 |
| 32k | 7,751 | **8,639** | 5,082 |
| 48k | 8,046 | **9,096** | 5,693 |
| 64k | 7,971 | **8,875** | 6,089 |
| 80k | 7,782 | **9,200** | 6,263 |
| 96k | 7,806 | **8,928** | 5,805 |
| 112k | 7,625 | **9,050** | 5,354 |
| 128k | 7,598 | **9,048** | 5,714 |
| 144k | 7,171 | **8,773** | 5,347 |
| 160k | 7,326 | **8,879** | 5,309 |
| 176k | 7,384 | **8,749** | 5,191 |
| 191k | 7,179 | **8,570** | 5,185 |
| 207k | 7,192 | **8,375** | 5,053 |
| 224k | 7,102 | **8,333** | 4,848 |
| 239k | 6,999 | **8,252** | 4,850 |
| 255k | 6,879 | **8,141** | 4,753 |

### Decode, tokens/s

| Context (tokens) | capy-flash | basalt | ninfer-ext |
|---|---|---|---|
| 16k | 178 | **235** | 79 |
| 32k | 149 | **212** | 87 |
| 48k | 164 | **236** | 99 |
| 64k | 158 | **242** | 85 |
| 80k | 150 | **227** | 69 |
| 96k | 137 | **187** | 84 |
| 112k | 132 | **186** | 87 |
| 128k | 130 | **207** | 83 |
| 144k | 145 | **203** | 84 |
| 160k | 133 | **178** | 72 |
| 176k | 155 | **214** | 64 |
| 191k | 150 | **169** | 77 |
| 207k | 161 | **178** | 77 |
| 224k | 145 | **178** | 57 |
| 239k | 139 | **146** | 69 |
| 255k | 143 | **163** | 65 |

## Summary across all 16 points

| Engine | Prefill tok/s | Decode greedy tok/s | Decode sampled tok/s | Sampled vs greedy decode |
|---|---|---|---|---|
| basalt | 8,285 - 9,211 | 176 - 265 | 146 - 242 | -7.1% |
| capy-flash | 6,828 - 8,015 | 138 - 198 | 130 - 178 | -7.7% |
| ninfer-ext | 4,473 - 6,269 | 73 - 106 | 57 - 99 | -9.8% |

## MTP draft acceptance, mean over the 16 points

| Engine | Greedy | Sampled | Change |
|---|---|---|---|
| basalt | 0.74 | 0.67 | -0.08 |
| capy-flash | 0.57 | 0.49 | -0.07 |
| ninfer-ext | 0.56 | 0.44 | -0.12 |

## Headline

* basalt is fastest at every context depth in both modes: 8.1k-9.2k tok/s prefill and 176-265 tok/s decode greedy.
* capy-flash is second: 6.8k-8.0k tok/s prefill, 138-198 tok/s decode greedy.
* ninfer-ext is third: 4.2k-6.3k tok/s prefill, 73-106 tok/s decode greedy.
* Decode ranks exactly with MTP draft acceptance: basalt 0.74, capy-flash 0.57, ninfer-ext 0.56 under greedy.
* Sampling costs 7-10% decode for every engine and nothing in prefill.

## Tuning that changed the result

| Engine | Change | Decode at 33k / 131k (tok/s) |
|---|---|---|
| basalt | stock auto picked pool-workers 1 on this CPU | 122 / 114 |
| basalt | pool-workers 15 | 238 / 200 |
| basalt | plus ple-io ram | 248 / 228 |
| capy-flash | stock profile | 175 / 138 |
| capy-flash | plus miss-split auto and 16 CPU threads | 184 / 173 |
| ninfer-ext | int8 KV (nvfp4 KV measured slower) | 106 / 83 |
```



document fetched 2026-10-23 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xcp0mf/qwen38fn_at_nvfp4_basalt_vs_cachyflash_vs
