# [GPU Lawyer] Deepseek 4.1 Flash 1018 on 4 32GB V100s -- pushing 40 tok/s decode, and will test across 8 v100s next. — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/8861bd3b7401319a

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-02 23:05 ingested 2026-11-03 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xmcibh/gpu_lawyer_deepseek_41_flash_1018_on_4_32gb_v100s/)

## retained text



```
[My SXM2 Game is Impeccable. I overcome obstacles. This room temperature is always tropical. ](https://preview.redd.it/lt0ldzxy5azh1.jpeg?width=1500&format=pjpg&auto=webp&s=32a33619ba58c1585b09ceae76d5a3968eff33d3)

TL;DR: I'm running DeepSeek-V4.1-Flash for a single user on four 2017-era V100 32 GB cards plus system RAM. Generation runs 36-38 tok/s from 8K to 75K context with thinking off, and 32-38 tok/s with thinking on. Prefill is about 1,000-1,200 tok/s from 4K to 128K depth. The context window is set to 524,288 tokens. It's built on the 1Cat-vLLM fork (at its #1277 merge), which does the hard Volta (sm70) work. I just built, tuned and benchmarked it.

Hardware

CPU: AMD Threadripper PRO 3945WX (12C/24T), single NUMA node

RAM: 256 GB DDR4 (8x 32 GB), run at 2667 MT/s for these tests

GPUs: 4x Tesla V100 SXM2 32 GB on one SXM2 carrier board

NVLink between all four (2 links per pair)

Host side: PCIe Gen3 x8 per GPU, behind a PLX switch with a single Gen3 x16 uplink to the CPU

Clocks/power: SM clock locked at 1522 MHz. Power limit 150 W per card, raised to 175 W for the API-speed and llama-benchy runs

Storage: the main GGUF loads from a SATA 8 TB hard drive. That only affects boot (about 22 minutes to load). The Engram file is on NVMe and gets pre-read into page cache.

Software / config

1Cat-vLLM at the #1277 merge (vLLM 1.5.2.dev1277), Python 3.12, PyTorch 2.10 + CUDA 12.8, NVIDIA driver 580

Tensor parallel 4, max-num-seqs 1 (single user), full CUDA graphs (batch sizes 1-6), async scheduling, prefix caching, chunked prefill, custom all-reduce over NVLink

Model: \`DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2\_XXS\` GGUF, 4 shards, \~232 GB

Engram table: \`DeepSeek-V4.1-Flash-Engram-IQ4\_NL\` GGUF (110.6 GB), kept in system RAM

Experts: about a third of the expert weights live in pinned host RAM (\`expert\_cache\_offload\_fraction: 0.33\`) and are swapped onto the GPUs on demand

Speculative decoding: built-in DSpark drafter, FP8, 5 draft tokens

Thinking on by default; tool calling with the \`deepseek\_v41\` parser

max-model-len 524288

Results

API decode speed (single request, unique cache salt per request, so no prefix-cache hits)

Test	Prompt	Decode tok/s (median of 3, range)	TTFT	Prefill tok/s	Accepted tokens per draft

8K, thinking off	\~8.0K	38.3 (36.6-39.0)	6.6 s	1,202	2.27

8K, thinking on	\~8.0K	37.9 (36.8-39.5)	6.5 s	1,234	1.39

75K, thinking off	\~75K	35.8 (33.2-38.1)	69.1 s	1,085	2.21

75K, thinking on	\~75K	31.9 (31.4-34.9)	69.2 s	1,084	1.34

8K, Engram file not yet in page cache	\~8.0K	25.9 (n=1)	9.9 s	808	2.34

The last row shows why it's worth pre-reading the 110 GB Engram file into RAM after boot.

llama-benchy 0.4.0: prefill (pp2048, concurrency 1, \`--no-cache\`, thinking off, 3 runs each)

Context depth	pp2048 t/s	Time to first response

0	578	4.0 s

4K	1,224	5.5 s

8K	1,014	10.6 s

16K	1,093	17.3 s

32K	1,117	31.6 s

64K	1,095	62.2 s

128K	1,008	132.5 s

llama-benchy 0.4.0: generation (pp2048 + tg, concurrency 1)

Context depth	tg128 t/s	tg1024 t/s

0	28.5	24.9

1K	24.8	31.2

4K	28.9	20.0

With a 5-token drafter, generation speed depends on how well the drafter guesses. llama-benchy's prompts are random book text, so tg1024 swings ±6-8 t/s from run to run. I left out benchy's tg32 numbers (13-26 t/s): 32 tokens is too short to measure anything with speculative decoding.

Engine-level numbers (offline, no HTTP)

Warm decode round (every expert the round needs is already on the GPUs): 27.8 ms, against 24.9 ms in upstream's reference. In the profiler a round spans 29.5 ms, of which the GPUs are busy 24.9 ms. That matches upstream, so the kernels perform as expected. The extra \~5 ms is host-side: CPU work at the start of each round, a blocking device-to-host copy, and small kernel-launch gaps. Newer upstream commits target exactly this per-step host overhead.

Fresh text (expert-cache misses): 116.7 ms/round and 25.0 tok/s, against upstream's 53.5 ms and 54.5 tok/s. Caveat: I ran this with the Engram file cold, so it isn't apples to apples. A warm rerun is on my list.

Host-to-GPU copies: 6.5 GB/s per GPU on its own, but 12.9 GB/s total (3.2 GB/s each) when all four copy at once, because they share one Gen3 x16 uplink. In the trace, all four GPUs are swapping at the same moment about 76% of the swap time.

Correctness

Greedy (temperature 0) output is identical run to run within one load (8 prompts x 2).

After a restart, 13 of 16 greedy outputs matched exactly. The 3 that diverged flipped at near-ties (top-two margins of 0.06-0.375 nats). I haven't tracked down the cause yet.

Thinking on/off and tool calls work.

All 8 upstream micro-tests pass (two needed hard-coded paths in the test files changed).

What's next

PCIe switch change: move to two switches so each Gen3 x16 uplink feeds 2 GPUs instead of 4. That roughly doubles host-to-GPU bandwidth when all cards swap experts at once. My estimate from the trace is a modest \~1.5 ms/round (\~5%) on warm text; it should matter more on fresh text.

RAM to 2933 MT/s (it was dropped from 3200 after memory errors).

Pull the newer 1Cat commits (per-step host fixes and DeepSeek-V4.1 updates) and re-run the same suite.

Move the GGUF from the hard drive to NVMe for faster boots.

Credits

All credit for the Volta DeepSeek-V4.1 path (GGUF experts, expert swapping, DSpark drafting on sm70) goes to the 1Cat-vLLM project (GitHub: yangzhuxinyzx and contributors). Happy to answer questions or share the full config.
```



document fetched 2026-11-03 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xmcibh/gpu_lawyer_deepseek_41_flash_1018_on_4_32gb_v100s
