# NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090 — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/f8de2893916b3f95

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-09-05 14:20 ingested 2026-09-06 02:41

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1w821fg/ninfer_vs_llamacpp_vs_vllm_quality_speed/)

## retained text



```
I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.

I was on llama.cpp (Q5\\\_K\\\_M GGUF, q5\\\_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.

# Hardware

\- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)  
\- Ryzen 7 7840HS, 32 GB DDR5  
\- Ubuntu 26.04, nvidia driver 610.43.02 (open)

**## Engine configs**

| | **\*\*llama.cpp\*\*** | **\*\*vLLM\*\*** | **\*\*NInfer\*\*** |  
|---|---|---|---|  
| Quant | Q5\\\_K\\\_M GGUF | NVFP4 | NVFP4 |  
| KV cache | q8\\\_0 | FP8 | FP8 |  
| Context | 196K | 262K | 240K |  
| MTP | On (gate failed) | None | MTP3 (76% acceptance) |  
| Concurrency | parallel=1 | Continuous batch | x2 lanes |  
| VRAM | 31.6 GB | 29.6 GB | 30.5 GB |

# How the eval worked

I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):

1. **\*\*Relevance classification\*\*** \- is this industry relevant? (binary, 50 labeled deals)  
2. **\*\*Needle retrieval\*\*** \- planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context  
3. **\*\*Multi-transcript QA\*\*** \- questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls  
4. **\*\*Reasoning with thinking\*\*** \- numeric/logic problems, thinking mode on, greedy pass@1  
5. **\*\*Structured extraction\*\*** \- custom extraction prompt, json\\\_mode (skipped on NInfer, it doesn't support json\\\_mode)  
6. **\*\*Tool replay\*\*** \- replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)

Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache\\\_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.

And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.

# Quality results

| **\*\*Tier\*\*** | **\*\*llama.cpp\*\*** | **\*\*vLLM\*\*** | **\*\*NInfer\*\*** |  
|---|---|---|---|  
| Relevance | 86.0% | 84.0% | 86.0% |  
| Needle (conditional) | 100% (29/29) | 100% (41/41) | 100% (41/41) |  
| Transcript QA | 82.0% | 78.0% | 88.0% |  
| Reasoning | 100% | 100% | 98.0% |  
| Extraction | F1 0.300 | F1 0.350 | skipped |  
| Tool replay | 0% | all errors | 0% |

Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max\\\_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.

Statistical comparison (NInfer vs llama.cpp, bootstrap):

\- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality  
\- Transcript QA: delta = -0.03, p=0.69 - no difference  
\- Reasoning: delta = +0.02, p=0.72 - no difference  
\- Relevance: McNemar p=1.0 - identical  
\- Tool replay: delta = 0.0 - both fail equally

**\*\*Takeaway: quality is statistically indistinguishable across all engines.\*\***

# Speed results (perf probe, server-side timings)

| **\*\*Metric\*\*** | **\*\*llama.cpp\*\*** | **\*\*NInfer\*\*** | **\*\*Speedup\*\*** |  
|---|---|---|---|  
| **\*\*Decode 1K\*\*** | 114 tok/s | 158 tok/s | 1.4x |  
| **\*\*Decode 32K\*\*** | 109 tok/s | 213 tok/s | 2.0x |  
| **\*\*Decode 128K\*\*** | 72 tok/s | 202 tok/s | **\*\*2.8x\*\*** |  
| Prefill 1K | 1,545 tok/s | 7,265 tok/s | **\*\*4.7x\*\*** |  
| Prefill 32K | 2,155 tok/s | 6,892 tok/s | 3.2x |  
| Prefill 128K | 1,528 tok/s | 3,904 tok/s | 2.6x |  
| TTFT 1K | 670 ms | 138 ms | 4.9x |  
| TTFT 32K | 15.2 s | 4.8 s | 3.2x |  
| TTFT 128K | 85.9 s | 33.6 s | 2.6x |

vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (\~66 tok/s). The speed difference is entirely MTP3 speculative decoding.

# What I learned

**\*\*NInfer's speed advantage is all MTP.\*\*** The raw NVFP4 kernel speed is about the same between NInfer and vLLM (\~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close.

**\*\*The decode speedup grows with context.\*\*** At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows.

**\*\*NInfer's tokenizer endpoint is great.\*\*** It exposes \`/v1/messages/count\_tokens\` (Anthropic Messages format) which gives exact token counts. No more \`len(text)//3\` heuristics.

**\*\*NInfer does NOT support json\\\_mode (as far as I can tell).\*\*** \`response\_format: json\_object\` returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement.

**\*\*Don't trust vibes for quality.\*\*** I went in expecting NVFP4 might lose a few points vs Q5\\\_K\\\_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE \~5.6pp).

# Verdict

NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json\\\_mode.

I put together a detailed poster with all the charts and methodology details: \[full results poster\](https://claude.ai/code/artifact/b6041437-2cf6-4198-a722-9b4ce853ccc3)

Setup if you want to try it:

\`\`\`  
\# NInfer (from source)  
git clone https://github.com/Neroued/ninfer && cd ninfer  
mkdir build && cd build  
cmake .. -DCMAKE\_BUILD\_TYPE=Release -GNinja && ninja

\# Model (HuggingFace)  
\# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB)

\# Run  
./ninfer-serve /path/to/model.ninfer \\  
  \--model-id qwen3.8-27b \\  
  \--host 0.0.0.0 --port 8080 \\  
  \--max-context 240000 --kv-capacity 240000 \\  
  \--max-concurrency 2 --kv-dtype fp8 \\  
  \--spec mtp --draft-tokens 3 \\  
  \--vision --preserve-thinking  
\`\`\
```



document fetched 2026-09-06 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1w821fg/ninfer_vs_llamacpp_vs_vllm_quality_speed
