← sources

Qwen3.8 27B on one RTX 5090: comparing six engines across TTFT, prefill, decode, concurrency and tool tasks

source: reddit (reddit:LocalLLaMA)published 2026-10-27 14:59ingested 2026-10-28 02:38

retained text

Disclosure: I’m the author of TensorSharp. This English post was prepared with LLM assistance from the benchmark report linked below.

Here is a comparison of **TensorSharp, llama.cpp, vLLM, SGLang, velox-inferer, and NInfer** running Qwen3.8 27B on a single RTX 5090.

The results cover short questions, long-document lookup, code and prose generation, passage copying, multi-turn chat, screenshot input, JSON output, tool calls, and concurrent requests.

The rankings change with the workload: NInfer leads speculative decode and four-request throughput, llama.cpp records the lowest short-prompt TTFT, and TensorSharp records the lowest completion time in the four-turn local tool test.

[Full report: configurations, model revisions, methodology, and results](https://claude.ai/artifact/2Jdw2gGRLbveNYMbhFnURN#2587fc59-b43b)

# Test setup

* **GPU:** RTX 5090, 32 GB VRAM, 575 W.
* **Host:** AMD EPYC 9354, 64 threads, Ubuntu 24.04, 93 GB system-memory limit.
* One engine running on the GPU at a time.
* Greedy decoding, temperature 0, **thinking disabled**, vision enabled.
* 40,960-token context configured per sequence.
* Three warm-up requests, followed by three measured repetitions per scenario.
* DFlash2 configured for seven draft tokens per step.
* Unique prompt nonces prevented reuse across independent requests. Multi-turn chat and the tool loop intentionally reused prefixes.

TensorSharp used its direct-CUDA backend, `--backend cuda`.

**The model configurations are not identical:**

* **TensorSharp / llama.cpp:** the same NVFP4 GGUF target and Q4\_K\_M drafter.
* **vLLM / SGLang:** NVIDIA’s NVFP4/FP8 safetensors target and an FP8 drafter.
* **NInfer / velox:** a mixed NVFP4/FP8 `.ninfer` target and Q8 drafter.

KV-cache formats also differ. These are results for complete configurations, including their quantization and memory requirements.

Throughout the tables, `plain → DFlash2` means without speculation → with speculation. “velox” means velox-inferer.

# 1. TTFT by task

**Client-observed time to first token, in seconds. Lower is better.**

TTFT includes HTTP handling, templating, tokenization, and prefill.

|Task|TensorSharp|llama.cpp|vLLM|SGLang|velox|NInfer|
|:-|:-|:-|:-|:-|:-|:-|
|Short factual/arithmetic question, \~50 tokens|0.050 → 0.064|0.027 → 0.030|0.061 → 0.136|0.042 → 0.048|0.088 → 0.088|0.051 → 0.053|
|4K document lookup, **plain only**|0.430|0.524|0.368|0.320|0.533|0.322|
|16K document lookup|1.64 → 1.69|2.05 → 3.51|1.45 → 1.59|1.34 → 1.38|1.80 → 1.92|1.24 → 1.26|
|32K document lookup|3.96 → 4.07|5.01 → 8.16|3.73 → 3.89|3.58 → 3.68|4.21 → 4.39|3.02 → 3.07|
|Multi-turn chat, turns 2–3|0.052 → 0.059|0.038 → 0.041|0.145 → 0.406|0.058 → 0.059|0.081 → 0.127|0.046 → 0.047|
|Screenshot input, 1344×768|0.125 → 0.142|0.309 → 0.326|0.127 → 0.259|0.153 → 0.202|0.250 → 0.209|0.162 → 0.163|

The document tasks retrieve a code word hidden in prose. Their approximate input lengths are **3.9K, 14.7K, and 31.8K tokens**. The report does not show the 4K DFlash2 result in its tables.

The multi-turn task uses an approximately 2.9K-token system prompt with deliberate prefix reuse. The screenshot contains approximately 1,000 visual tokens.

What stands out:

* **Short questions:** llama.cpp records **27 ms plain / 30 ms speculative**.
* **Long inputs:** NInfer records the lowest 16K and 32K TTFT.
* **Speculation overhead:** llama.cpp’s 32K TTFT increases from **5.01 to 8.16 seconds**, while the other configurations add less than 0.2 seconds.
* **Image input:** TensorSharp’s **125 ms** and vLLM’s **127 ms** plain results are effectively tied. With DFlash2, TensorSharp records the lowest measured screenshot TTFT at **142 ms**.

These screenshot timings measure the complete image-request path before the first output token.

# 2. Prefill tokens/sec

The report directly gives these **plain-mode prefill ranges** across the long-input tests:

* **TensorSharp:** approximately **8.0–9.3K tok/s**.
* **llama.cpp:** approximately **6.3–7.6K tok/s**.
* **NInfer and SGLang, reported together:** approximately **8.9–12.4K tok/s**.

It does not provide a complete per-engine, per-task table of isolated prefill throughput.

For a six-engine comparison on the 32K document task, the following values are **estimated effective input throughput**:

`estimated input tok/s = approximately 31,800 input tokens ÷ reported client TTFT`

|Input task and mode|TensorSharp|llama.cpp|vLLM|SGLang|velox|NInfer|
|:-|:-|:-|:-|:-|:-|:-|
|32K document lookup, plain|≈8.0K|≈6.3K|≈8.5K|≈8.9K|≈7.6K|≈10.5K|
|32K document lookup, DFlash2|≈7.8K|≈3.9K|≈8.2K|≈8.6K|≈7.2K|≈10.4K|

**All values in this table are derived estimates in input tokens/sec.** They include request-processing overhead and use rounded prompt lengths and median TTFT values. They are not isolated GPU prefill-kernel measurements.

# 3. Decode tokens/sec by generation task

**Output tokens/sec. Higher is better.**

The individual task rows use **DFlash2**. The final row compares the report’s three-workload average with and without speculation.

|Generation task or summary|TensorSharp|llama.cpp|vLLM|SGLang|velox|NInfer|
|:-|:-|:-|:-|:-|:-|:-|
|Code generation, DFlash2|339|321|383|368|346|415|
|Prose generation, DFlash2|174|160|189|184|184|209|
|Passage copying, DFlash2|190|158|171|152|175|217|
|Three-workload average, **plain → DFlash2**|80.0 → 234.4|84.2 → 212.9|69.4 → 247.8|78.5 → 234.5|68.8 → 234.9|77.0 → 280.5|

Decode rate is calculated as:

`(completion tokens − 1) ÷ (last-token time − first-token time)`

The summary is the mean of the code, prose, and copying scenario medians, rather than an average across all benchmark tasks. Individual task values are rounded. The report does not separately show plain-mode decode rates for each generation task.

NInfer leads all three speculative-generation tasks. TensorSharp, SGLang, and velox have similar averages but different workload profiles:

* SGLang reaches **368 tok/s on code**, compared with TensorSharp’s **339**.
* TensorSharp reaches **190 tok/s on copying**, compared with SGLang’s **152**.
* Coding rates are substantially higher than prose rates across all six engines.

DFlash2 improves the summarized single-request decode rate by approximately **2.5–3.6×**.

# 4. Parallel requests: throughput and latency

These tests submit independent requests together. **Throughput is total output tokens/sec across the submitted requests.**

|Workload and metric|TensorSharp|llama.cpp|vLLM|SGLang|velox|NInfer|
|:-|:-|:-|:-|:-|:-|:-|
|2 requests, plain: total tok/s|137|154|116|140|67|144|
|4 requests: total tok/s, plain → DFlash2|215 → 445|251 → 259|226 → 304|148 → 251\*|67 → 303|257 → 488|
|4 requests: TTFT in seconds, plain → DFlash2|0.119 → 0.147|0.093 → 0.126|0.243 → 0.265|0.091 → 1.06\*|6.19 → 0.475|0.156 → 0.139|

**\* SGLang’s DFlash2 configuration fit only two active request lanes in 32 GB.** Four requests were submitted, so some queued. This affects both throughput and latency.

vLLM’s speculative configuration required `--gpu-memory-utilization 0.97`. The report does not show the two-request DFlash2 throughput in its summary tables.

NInfer leads four-request speculative throughput at **488 tok/s**, followed by TensorSharp at **445 tok/s**.

velox’s plain configuration served requests sequentially, while its speculative configuration overlapped them. Its reported four-request TTFT changes from **6.19 to 0.475 seconds**.

The concurrency workload differs from the single-request generation tasks, so these numbers should not be used to calculate batching efficiency against the earlier decode averages.

# 5. JSON and tool-use tasks

**End-to-end completion time in seconds, plain → DFlash2. Lower is better.**

|Task|TensorSharp|llama.cpp|vLLM|SGLang|velox|NInfer|
|:-|:-|:-|:-|:-|:-|:-|
|JSON object|0.911 → 0.281|0.839 → 0.314|1.11 → 0.391|0.915 → 0.304|1.13 → 0.369|0.579 → 0.274†|
|Strict JSON schema|1.43 → 0.309|1.38 → 0.473|1.68 → 0.445|1.41 → 0.360|1.81✗ → 0.432✗|0.826 → 0.258|
|One function call|0.563 → 0.199|0.511 → 0.218|0.699 → 0.354|0.535 → 0.171|0.771 → 0.383|0.571 → 0.180|
|Three function calls in one response|1.63 → 0.427|1.53 → 0.484|1.92 → 0.539|1.62 → 0.457|2.02 → 0.680|1.68 → 0.378|
|Four-turn local tool loop|2.47 → 0.748|2.15 → 0.817|2.98 → 1.45|2.31 → 0.791|2.93 → 1.42|2.71 → 0.817|

**Correctness matters:**

* **✗:** velox failed the JSON-schema case in all three repetitions in both modes.
* **†:** NInfer with DFlash2 produced an empty JSON key in one of three JSON-object repetitions and failed that check.
* The other functional checks passed.

JSON response lengths differed. The report notes that NInfer’s plain JSON replies were approximately **40% shorter**, which contributes to its completion-time advantage.

The tool results also change by operation:

* **One call:** llama.cpp has the lowest plain time, **0.511 s**; SGLang has the lowest speculative time, **0.171 s**.
* **Three calls:** llama.cpp leads plain mode at **1.53 s**; NInfer leads with DFlash2 at **0.378 s**.
* **Four-turn loop:** llama.cpp leads plain mode at **2.15 s**; TensorSharp records the lowest observed speculative time at **0.748 s**.

The loop uses locally answered file-listing, file-reading, and calculator tools. It does not include browser actions or external-service latency. Three function calls emitted in one response are also different from three concurrent HTTP requests.

Some engines buffer complete tool calls or structured replies before streaming. For those responses, the first visible output arrives at completion, so TTFT equals end-to-end time.

# Limits and follow-up work

These results come from one GPU and three measured repetitions per scenario. The report observed up to roughly **4% decode variation** between runs, so small gaps should be treated cautiously.

The quantized targets, drafters, and cache formats differ. Passing these functional checks does not establish equivalent model quality.

Thinking mode, non-greedy sampling, concurrency above four, prompts beyond 32K, video, and systematic memory-use measurements remain outside this comparison.

For TensorSharp, the four-request throughput and local tool-loop results are encouraging. The single-request results also show where further optimization would be useful, especially the DFlash2 path relative to NInfer and vLLM.

If you run one of these engines on a 5090 and see a configuration choice that materially disadvantages it, I’d appreciate the specific setting you would change and why.

[TensorSharp source](https://github.com/zhongkaifu/TensorSharp)

document fetched 2026-10-28 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xguu4c/qwen38_27b_on_one_rtx_5090_comparing_six_engines