# (NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/dc96dca4b2798fed

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-08-29 18:24 ingested 2026-08-30 02:40

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1w1txyk/ninfer_fork_i_wanted_to_have_a_1m_context_qwen38/)

## retained text



```
Hey! I forked [NInfer](https://github.com/Neroued/ninfer) (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 run a 1,048,576-token context on two consumer 5090s — 27.4 GB per card, no NVLink.

Numbers (single stream, 500 W per GPU cap):



* Decode at 653k context: 119 tok/s with MTP speculative decoding, 57 without. vLLM on the same prompts: 42 tok/s.
* Why: past its native 262k window, vLLM's MTP acceptance drops to exactly zero (0 of 1,533 drafts accepted) — it keeps paying for the drafter. NInfer's acceptance stays \~55–60% out to 1M.
* Prefill is where vLLM wins: 1.2–1.3× faster. A full 1M prompt takes \~18 minutes on NInfer. That's the untuned part.
* At 1M: 48 tok/s decode, \~100 with MTP.
* vLLM's fp8 KV cache tops out at \~759k tokens on this hardware; NInfer's INT8 KV fits 1,048,576 in less memory.



Two GPUs are also just faster than one: 75 vs 54 tok/s at 250k, because weights and KV traffic halve per card and the \~128 cross-GPU reductions per token cost only \~0.2 ms under CUDA graphs.

  
[Fork](https://github.com/wamansou/ninfer-tp2-1m)

[Performance ](https://github.com/wamansou/ninfer-tp2-1m/blob/master/docs/performance.md)


```



document fetched 2026-08-30 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1w1txyk/ninfer_fork_i_wanted_to_have_a_1m_context_qwen38
