← sources

28 TPS on Qwen2.5-7B across two separate cloud regions over public WAN using speculative decoding + CUDA Graphs [P]

source: reddit (reddit:MachineLearning)published 2026-08-23 12:30ingested 2026-08-24 06:56

retained text

<!-- SC_OFF --><div class="md"><p>been building ShardFlow for the past few months, a distributed LLM inference </p> <p>framework that splits any HuggingFace transformer across N GPU machines and uses </p> <p>neural speculative decoding to deal with WAN latency.</p> <p>the setup for the benchmark: two T4 nodes in separate GCP regions (Iowa + Oregon) </p> <p>talking through an AWS EC2 TCP relay in Ohio. ~86ms RTT on public internet.</p> <p>the key insight with speculative decoding here is that WAN latency stops being a </p> <p>per-token cost and becomes a per-round cost. with K=8 drafting you're committing </p> <p>4.07 tokens per round trip instead of 1. at 86ms RTT that's a big deal.</p> <p>numbers on Qwen2.5-7B:</p> <p>non-speculative baseline: 4.92 TPS</p> <p>neural drafter (eager): 14.3 TPS peak</p> <p>+ CUDA Graphs on drafter: 28.10 TPS peak / 20.31 TPS avg</p> <p>also ran Qwen2.5-14B with NF4 4-bit quant, same two nodes: 14.43 TPS avg.</p> <p>the v2.1 fix that surprised me most: draft generation was launching ~1500 CUDA </p> <p>kernels per round from a Python loop. each kernel 2-5us, Python launch overhead </p> <p>8-10us. GPU sitting idle 65% of the time. capturing the full 0.5B forward pass </p> <p>as a CUDA Graph and replaying with one driver call dropped draft latency from </p> <p>112ms to 25ms.</p> <p>other things in the stack: zero-copy Rust TCP relay, StaticCache + in-place KV </p> <p>rewind for graph compatibility, meta-device model slicing to avoid loading 15GB </p> <p>into CPU RAM.</p> <p>repo: <a href="https://github.com/rautaditya2606/Shardflow">https://github.com/rautaditya2606/Shardflow</a></p> <p>happy to answer questions on the speculative decoding implementation or the CUDA </p> <p>graphs stuff specifically.</p> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/katua_bkl"> /u/katua_bkl </a> <br /> <span><a href="https://www.reddit.com/r/MachineLearning/comments/1vw5ysj/28_tps_on_qwen257b_across_two_separate_cloud/">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/MachineLearning/comments/1vw5ysj/28_tps_on_qwen257b_across_two_separate_cloud/">[comments]</a></span>

document fetched 2026-08-24 06:56 · canonical https://reddit.com/r/MachineLearning/comments/1vw5ysj/28_tps_on_qwen257b_across_two_separate_cloud