NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090
retained text
I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090. I was on llama.cpp (Q5\\\_K\\\_M GGUF, q5\\\_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes. # Hardware \- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading) \- Ryzen 7 7840HS, 32 GB DDR5 \- Ubuntu 26.04, nvidia driver 610.43.02 (open) **## Engine configs** | | **\*\*llama.cpp\*\*** | **\*\*vLLM\*\*** | **\*\*NInfer\*\*** | |---|---|---|---| | Quant | Q5\\\_K\\\_M GGUF | NVFP4 | NVFP4 | | KV cache | q8\\\_0 | FP8 | FP8 | | Context | 196K | 262K | 240K | | MTP | On (gate failed) | None | MTP3 (76% acceptance) | | Concurrency | parallel=1 | Continuous batch | x2 lanes | | VRAM | 31.6 GB | 29.6 GB | 30.5 GB | # How the eval worked I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks): 1. **\*\*Relevance classification\*\*** \- is this industry relevant? (binary, 50 labeled deals) 2. **\*\*Needle retrieval\*\*** \- planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context 3. **\*\*Multi-transcript QA\*\*** \- questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls 4. **\*\*Reasoning with thinking\*\*** \- numeric/logic problems, thinking mode on, greedy pass@1 5. **\*\*Structured extraction\*\*** \- custom extraction prompt, json\\\_mode (skipped on NInfer, it doesn't support json\\\_mode) 6. **\*\*Tool replay\*\*** \- replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one) Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache\\\_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins. And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran. # Quality results | **\*\*Tier\*\*** | **\*\*llama.cpp\*\*** | **\*\*vLLM\*\*** | **\*\*NInfer\*\*** | |---|---|---|---| | Relevance | 86.0% | 84.0% | 86.0% | | Needle (conditional) | 100% (29/29) | 100% (41/41) | 100% (41/41) | | Transcript QA | 82.0% | 78.0% | 88.0% | | Reasoning | 100% | 100% | 98.0% | | Extraction | F1 0.300 | F1 0.350 | skipped | | Tool replay | 0% | all errors | 0% | Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max\\\_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit. Statistical comparison (NInfer vs llama.cpp, bootstrap): \- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality \- Transcript QA: delta = -0.03, p=0.69 - no difference \- Reasoning: delta = +0.02, p=0.72 - no difference \- Relevance: McNemar p=1.0 - identical \- Tool replay: delta = 0.0 - both fail equally **\*\*Takeaway: quality is statistically indistinguishable across all engines.\*\*** # Speed results (perf probe, server-side timings) | **\*\*Metric\*\*** | **\*\*llama.cpp\*\*** | **\*\*NInfer\*\*** | **\*\*Speedup\*\*** | |---|---|---|---| | **\*\*Decode 1K\*\*** | 114 tok/s | 158 tok/s | 1.4x | | **\*\*Decode 32K\*\*** | 109 tok/s | 213 tok/s | 2.0x | | **\*\*Decode 128K\*\*** | 72 tok/s | 202 tok/s | **\*\*2.8x\*\*** | | Prefill 1K | 1,545 tok/s | 7,265 tok/s | **\*\*4.7x\*\*** | | Prefill 32K | 2,155 tok/s | 6,892 tok/s | 3.2x | | Prefill 128K | 1,528 tok/s | 3,904 tok/s | 2.6x | | TTFT 1K | 670 ms | 138 ms | 4.9x | | TTFT 32K | 15.2 s | 4.8 s | 3.2x | | TTFT 128K | 85.9 s | 33.6 s | 2.6x | vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (\~66 tok/s). The speed difference is entirely MTP3 speculative decoding. # What I learned **\*\*NInfer's speed advantage is all MTP.\*\*** The raw NVFP4 kernel speed is about the same between NInfer and vLLM (\~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close. **\*\*The decode speedup grows with context.\*\*** At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows. **\*\*NInfer's tokenizer endpoint is great.\*\*** It exposes \`/v1/messages/count\_tokens\` (Anthropic Messages format) which gives exact token counts. No more \`len(text)//3\` heuristics. **\*\*NInfer does NOT support json\\\_mode (as far as I can tell).\*\*** \`response\_format: json\_object\` returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement. **\*\*Don't trust vibes for quality.\*\*** I went in expecting NVFP4 might lose a few points vs Q5\\\_K\\\_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE \~5.6pp). # Verdict NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json\\\_mode. I put together a detailed poster with all the charts and methodology details: \[full results poster\](https://claude.ai/code/artifact/b6041437-2cf6-4198-a722-9b4ce853ccc3) Setup if you want to try it: \`\`\` \# NInfer (from source) git clone https://github.com/Neroued/ninfer && cd ninfer mkdir build && cd build cmake .. -DCMAKE\_BUILD\_TYPE=Release -GNinja && ninja \# Model (HuggingFace) \# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB) \# Run ./ninfer-serve /path/to/model.ninfer \\ \--model-id qwen3.8-27b \\ \--host 0.0.0.0 --port 8080 \\ \--max-context 240000 --kv-capacity 240000 \\ \--max-concurrency 2 --kv-dtype fp8 \\ \--spec mtp --draft-tokens 3 \\ \--vision --preserve-thinking \`\`\`
document fetched 2026-09-06 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1w821fg/ninfer_vs_llamacpp_vs_vllm_quality_speed