← sources

Some "trust me bro" benchmarks while we prep the repo to open-source

source: reddit (reddit:LocalLLaMA)published 2026-11-03 22:27ingested 2026-11-04 02:41

retained text

A month ago we picked up a few IGX Thor boxes equipped with RTX PRO 6000 GPUs. I shared some numbers earlier running Qwen 3.8 27B on SGLang. These machines were bought specifically to run our upcoming on-device agent platform, steffi.ai.

Since the goal is local, multi-agent systems where models run strictly on-device, generic inference engines started hitting limits. Concurrency scaling, cold start/wake times, and turn-to-turn latency under massive context windows were making or breaking our agent loops.

So we started building a dedicated Rust-based inference engine, `llminfra`. After a few weeks of heads-down work, we wrapped the first internal release to run Qwen locally. We benchmarked it side-by-side against SGLang on the exact same hardware (RTX PRO 6000, 300W, identical workloads).

Here are the numbers from the run:

* **Turn TTFT on 250K context:** 108 ms vs 1,440 ms (13.4x faster). In agent loops where the context stays loaded and the agent just appends new tool call observations, cutting follow-up latency from 1.4s down to 100ms keeps multi-turn workflows from feeling sluggish.
* **Cold start to ready:** 6.1s vs 211.2s (34.8x faster). Spinning up or recovering a worker doesn't stall for 3+ minutes. First token after start clocked at 28 ms vs 528 ms.
* **Throughput under heavy agent load:** With 32 agents running 32K context, completion throughput hit 209 tok/s vs 95 tok/s (2.2x). At 2 agents pushing 250K context, it sustained 86 tok/s vs 28 tok/s (3.1x).
* **Peak VRAM:** 52.9 GiB vs 86.3 GiB on an 8x32K workload.

We are finishing up documentation, cleaning up build scripts, and stripping out proprietary hooks so we can get the repo public for everyone to poke at and benchmark on their own gear.

Full benchmark charts attached. Let me know what other breakdowns or workload scenarios you would like to see when the code goes live.

document fetched 2026-11-04 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xn78cm/some_trust_me_bro_benchmarks_while_we_prep_the