Qwen3.8-FN at NVFP4 ( Basalt vs cachy-flash vs ninfer-ext)
retained text
# Qwen3.8-Flash-Next NVFP4 on one RTX 5090 - prefill and decode against context Full 262,144-token context, one request at a time, MTP speculation on, int8 KV, identical test for all three engines. Every number is tokens/second. Prefill = prompt tokens / time to first token. Decode = generated tokens / decode time. The fastest engine in each row is in bold. Each data point is a fresh, non-repeating prompt: the servers reported zero prefix-cache hits. | Item | Value | |---|---| | GPU | NVIDIA RTX 5090, 32 GB | | CPU, RAM | Intel Core Ultra 9 285K, 247 GB DDR5 | | Model | Qwen3.8-Flash-Next, NVIDIA NVFP4 experts, 48 layers, 512 experts | | Artifacts | basalt `Qwen3.8-FN-NVFP4.basalt`, capy-flash `qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer`, ninfer-ext `qwen3_8_flash_next_nvfp4_draft.ninfer` | | Context | 262,144 tokens, swept in 16k steps (16 points) | | Output | 256 tokens per request, ignore_eos | | Builds | basalt `fe7fd96f`, capy-flash `4e45286a` v0.7.1, ninfer-ext `3241c511` | ## Engine configuration (after tuning) | Setting | basalt | capy-flash | ninfer-ext | |---|---|---|---| | max context | 262,144 | 262,144 | 262,144 | | KV cache | int8, elastic | int8, elastic | int8 | | expert cache | auto | auto (elastic) | auto | | speculation | MTP, spec window 7 | MTP, 3 drafts | MTP, 3 drafts | | CPU side | pool-workers 15, ple-io ram | miss-split auto, 16 CPU threads | ngram mapped | | VRAM used | 27.1 GiB + 40.7 GiB host RAM | 28.3 GiB + 65.2 GiB host | 23.1 GiB | ## Greedy decoding, temperature 0 ### Prefill, tokens/s | Context (tokens) | capy-flash | basalt | ninfer-ext | |---|---|---|---| | 16k | 6,921 | **8,490** | 4,473 | | 32k | 7,736 | **8,712** | 5,193 | | 48k | 8,015 | **9,211** | 5,764 | | 64k | 7,976 | **9,052** | 6,113 | | 80k | 7,806 | **9,206** | 6,269 | | 96k | 7,879 | **8,949** | 5,842 | | 112k | 7,800 | **9,061** | 5,353 | | 128k | 7,677 | **8,978** | 5,696 | | 144k | 7,238 | **8,842** | 5,321 | | 160k | 7,429 | **8,773** | 5,311 | | 176k | 7,391 | **8,670** | 5,182 | | 191k | 7,283 | **8,553** | 5,187 | | 207k | 7,207 | **8,511** | 5,057 | | 224k | 7,063 | **8,434** | 4,847 | | 239k | 6,912 | **8,347** | 4,846 | | 255k | 6,828 | **8,285** | 4,748 | ### Decode, tokens/s | Context (tokens) | capy-flash | basalt | ninfer-ext | |---|---|---|---| | 16k | 198 | **242** | 92 | | 32k | 186 | **265** | 96 | | 48k | 170 | **251** | 106 | | 64k | 160 | **246** | 99 | | 80k | 167 | **242** | 80 | | 96k | 157 | **205** | 87 | | 112k | 157 | **225** | 90 | | 128k | 165 | **216** | 88 | | 144k | 157 | **202** | 88 | | 160k | 151 | **187** | 87 | | 176k | 151 | **199** | 78 | | 191k | 162 | **196** | 83 | | 207k | 152 | **185** | 76 | | 224k | 151 | **187** | 74 | | 239k | 144 | **177** | 73 | | 255k | 138 | **176** | 75 | ## Sampled decoding, temperature 1.0, top_p 0.95, top_k 20 ### Prefill, tokens/s | Context (tokens) | capy-flash | basalt | ninfer-ext | |---|---|---|---| | 16k | 6,881 | **8,571** | 4,198 | | 32k | 7,751 | **8,639** | 5,082 | | 48k | 8,046 | **9,096** | 5,693 | | 64k | 7,971 | **8,875** | 6,089 | | 80k | 7,782 | **9,200** | 6,263 | | 96k | 7,806 | **8,928** | 5,805 | | 112k | 7,625 | **9,050** | 5,354 | | 128k | 7,598 | **9,048** | 5,714 | | 144k | 7,171 | **8,773** | 5,347 | | 160k | 7,326 | **8,879** | 5,309 | | 176k | 7,384 | **8,749** | 5,191 | | 191k | 7,179 | **8,570** | 5,185 | | 207k | 7,192 | **8,375** | 5,053 | | 224k | 7,102 | **8,333** | 4,848 | | 239k | 6,999 | **8,252** | 4,850 | | 255k | 6,879 | **8,141** | 4,753 | ### Decode, tokens/s | Context (tokens) | capy-flash | basalt | ninfer-ext | |---|---|---|---| | 16k | 178 | **235** | 79 | | 32k | 149 | **212** | 87 | | 48k | 164 | **236** | 99 | | 64k | 158 | **242** | 85 | | 80k | 150 | **227** | 69 | | 96k | 137 | **187** | 84 | | 112k | 132 | **186** | 87 | | 128k | 130 | **207** | 83 | | 144k | 145 | **203** | 84 | | 160k | 133 | **178** | 72 | | 176k | 155 | **214** | 64 | | 191k | 150 | **169** | 77 | | 207k | 161 | **178** | 77 | | 224k | 145 | **178** | 57 | | 239k | 139 | **146** | 69 | | 255k | 143 | **163** | 65 | ## Summary across all 16 points | Engine | Prefill tok/s | Decode greedy tok/s | Decode sampled tok/s | Sampled vs greedy decode | |---|---|---|---|---| | basalt | 8,285 - 9,211 | 176 - 265 | 146 - 242 | -7.1% | | capy-flash | 6,828 - 8,015 | 138 - 198 | 130 - 178 | -7.7% | | ninfer-ext | 4,473 - 6,269 | 73 - 106 | 57 - 99 | -9.8% | ## MTP draft acceptance, mean over the 16 points | Engine | Greedy | Sampled | Change | |---|---|---|---| | basalt | 0.74 | 0.67 | -0.08 | | capy-flash | 0.57 | 0.49 | -0.07 | | ninfer-ext | 0.56 | 0.44 | -0.12 | ## Headline * basalt is fastest at every context depth in both modes: 8.1k-9.2k tok/s prefill and 176-265 tok/s decode greedy. * capy-flash is second: 6.8k-8.0k tok/s prefill, 138-198 tok/s decode greedy. * ninfer-ext is third: 4.2k-6.3k tok/s prefill, 73-106 tok/s decode greedy. * Decode ranks exactly with MTP draft acceptance: basalt 0.74, capy-flash 0.57, ninfer-ext 0.56 under greedy. * Sampling costs 7-10% decode for every engine and nothing in prefill. ## Tuning that changed the result | Engine | Change | Decode at 33k / 131k (tok/s) | |---|---|---| | basalt | stock auto picked pool-workers 1 on this CPU | 122 / 114 | | basalt | pool-workers 15 | 238 / 200 | | basalt | plus ple-io ram | 248 / 228 | | capy-flash | stock profile | 175 / 138 | | capy-flash | plus miss-split auto and 16 CPU threads | 184 / 173 | | ninfer-ext | int8 KV (nvfp4 KV measured slower) | 106 / 83 |
document fetched 2026-10-23 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xcp0mf/qwen38fn_at_nvfp4_basalt_vs_cachyflash_vs