I turned an asymetric pair of Tesla V100s PCIe both (16 GB + 32 GB) into a surprisingly capable local LLM lab — 1.38k prompt tok/s, 40 decode tok/s with qwen3.8 27B Q6 and Q8...
retained text
#
**TL;DR:** I run a mismatched Tesla V100-PCIE pair—one 16 GB card and one 32 GB card, 48 GB total—in a Proxmox/LXC-based local-inference lab. The practical winner so far is a recent CUDA build of `llama.cpp` with tensor split, Flash Attention, `--numa distribute`, and large batches. On Qwen3.8 27B Q6\_K\_M I measured **1,376.9 prompt tok/s at 2k**, **1,324.3 prompt tok/s at 4k**, **1,221.5 prompt tok/s at 16k**, and **39.9 decode tok/s**. I also made Qwen3.8 Flash Next Q4 GGUF of a 177 B/6B-active MoE model run across the same two cards at roughly 26 tok/s decode and 380 PP.
This is not a claim that old V100s beat modern GPUs. It is a report of what they can do in this crazy "a kidney for a GPU" market!
My Hardware:
| Layer | Hardware / configuration |
| GPU 0 | NVIDIA Tesla V100-PCIE, 16,144 MiB VRAM, compute capability 7.0 |
| GPU1 | NVIDIA Tesla V100-PCIE, 32,494 MiB VRAM, compute capability 7.0 |
| Server topology | NUMA-aware host; benchmarks use `numactl --interleave=all` or `--numa distribute` |
| Virtualization | Proxmox with LXC inference containers for isolated runtimes |
| CPU| 2 x Xeon E5-2696 v4 @ 2.20GHz|
| RAM | 512 GB DDR4 2400 ECC RDIMM|
| CUDA stack | CUDA 12.8; NVIDIA driver libraries are host-mounted into the vLLM LXC where needed
|Workload|Runtime / model|Best result|Configuration behind the result|
|:-|:-|:-|:-|
|**Best 27B prefill**|Mainline llama.cpp, Qwen3.8 27B Q6\_K\_M|**1,376.87 pp tok/s at 2k**|Tensor split; main GPU = 32 GB V100; Flash Attention; Q8 KV; batch/ubatch 2,048; 20 threads; NUMA distribute|
|**Best 27B decode**|Mainline llama.cpp, Qwen3.8 27B Q6\_K\_M|**39.88 tg tok/s**|Same configuration|
|27B at 16k context|Mainline llama.cpp, Qwen3.8 27B Q6\_K\_M|**1,221.49 pp tok/s**|Same configuration|
|Best Q8 27B prefill|Mainline llama.cpp, Qwen3.8 27B Q8\_K\_XL|**1,059.67 pp tok/s at 2k**|Tensor split; long-context run used 1:1.5 split|
|Q8 27B at 64k|Mainline llama.cpp, Qwen3.8 27B Q8\_K\_XL|**664.36 pp tok/s**|65,536-token prompt; tensor split 1:1.5|
|Best ik\_llama 27B result|ik\_llama.cpp, Qwen3.8 27B Q6\_K\_M|**670.61 pp tok/s at 2k; 35.43 tg tok/s**|Graph split; 1:1 split; smaller batch settings than the mainline winner|
|Largest model tested|ik\_llama.cpp, Qwen3.8 Flash Next Q4\_K\_XL|**380.67 pp tok/s at 4k; 26.29 tg tok/s**|103.68 GiB GGUF; reported 177B total / 6B active MoE; graph split 1:1.75|
|ninfer-v100|NInfer, Qwen3.8 27B NVFP4|**927.54 pp tok/s at 8,190**|19.7 GiB weights; INT8 KV; MTP|
What the tuning showed
|Qwen3.8 27B Q6\_K\_M, mainline llama.cpp|pp2048|tg128|
|:-|:-|:-|
|Layer split, 79 threads|981.79|22.88|
|Tensor split, 20 threads|1,371.57|38.79|
|Tensor split, main GPU = 32 GB V100|**1,376.87**|**39.88**|
|Tensor split 1:2|980.71|26.10|
|No split|967.36|18.05|
**Takeaway:** on this uneven V100 pair, tensor split—not layer split—and using the 32 GB V100 as main GPU are decisive. The best daily-driver path is mainline llama.cpp + Qwen3.8 27B Q6 + tensor split + Flash Attention + Q8 KV + NUMA distribution.
root@llama-cpp:\~# numactl --interleave=all /opt/llama.cpp/build/bin/llama-bench -m /path/to/Qwen3.8-27B-UD-Q6\_K\_M.gguf -ngl 999 -sm tensor -mg 1 -fa on -b 2048 -ub 2048 -t 20 -p 512,2048,4096,16384 -n 128 -r
5 -lm none --numa distribute
ggml\_cuda\_init: found 2 CUDA devices (Total VRAM: 48638 MiB):
Device 0: Tesla V100-PCIE-16GB, compute capability 7.0, VMM: yes, VRAM: 16144 MiB
Device 1: Tesla V100-PCIE-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
| model | size | params | backend | ngl | threads | n\_ubatch | main\_gpu | sm | fa | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | ---------: | -----: | --: | ---------: | --------------: | -------------------: |
| qwen35 27B Q6\_K | 21.49 GiB | 27.32 B | CUDA | 999 | 20 | 2048 | 1 | tensor | 1 | none | pp512 | 1156.94 ± 5.10 |
| qwen35 27B Q6\_K | 21.49 GiB | 27.32 B | CUDA | 999 | 20 | 2048 | 1 | tensor | 1 | none | pp2048 | 1376.87 ± 4.59 |
| qwen35 27B Q6\_K | 21.49 GiB | 27.32 B | CUDA | 999 | 20 | 2048 | 1 | tensor | 1 | none | pp4096 | 1324.34 ± 7.37 |
| qwen35 27B Q6\_K | 21.49 GiB | 27.32 B | CUDA | 999 | 20 | 2048 | 1 | tensor | 1 | none | pp16384 | 1221.49 ± 3.87 |
| qwen35 27B Q6\_K | 21.49 GiB | 27.32 B | CUDA | 999 | 20 | 2048 | 1 | tensor | 1 | none | tg128 | 39.88 ± 0.04 |
build: b29c606 (1)
root@llama-cpp:\~# for MAIN in 0 1; do
echo
echo "===== MAIN GPU ${MAIN} ====="
/usr/bin/numactl --interleave=all \\
/opt/ik\_llama.cpp/build/bin/llama-bench \\
\-m /path/to/models/Qwen3.8-Flash-Next-UD-Q4\_K\_XL-00001-of-00004.gguf -ngl 999 -sm graph -ts 1/1.75 -mg "$MAIN" --fit 1 --fit-margin 1024 -fa 1 -p 4096,8192 -n 128 -b 2048 -ub 1024 -r 10 -o md
done 2>&1 | tee /root/bench-qwen38-main-gpu-final.md
===== MAIN GPU 0 =====
ggml\_cuda\_init: found 2 CUDA devices:
Device 0: Tesla V100-PCIE-16GB, compute capability 7.0, VMM: yes, VRAM: 16144 MiB
Device 1: Tesla V100-PCIE-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
| model | size | params | backend | ngl | n\_ubatch | sm | ts | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ----: | ------------ | ------------: | ---------------: |
| qwen4exp 125B.A6B Q4\_K - Medium | 103.68 GiB | 176.94 B | CUDA | 999 | 1024 | graph | 1.00/1.75 | pp4096 | 356.20 ± 7.68 |
| qwen4exp 125B.A6B Q4\_K - Medium | 103.68 GiB | 176.94 B | CUDA | 999 | 1024 | graph | 1.00/1.75 | pp8192 | 361.00 ± 4.72 |
| qwen4exp 125B.A6B Q4\_K - Medium | 103.68 GiB | 176.94 B | CUDA | 999 | 1024 | graph | 1.00/1.75 | tg128 | 26.44 ± 0.11
===== MAIN GPU 1 =====
ggml\_cuda\_init: found 2 CUDA devices:
Device 0: Tesla V100-PCIE-16GB, compute capability 7.0, VMM: yes, VRAM: 16144 MiB
Device 1: Tesla V100-PCIE-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
| model | size | params | backend | ngl | n\_ubatch | main\_gpu | sm | ts | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | ----: | ------------ | ------------: | ---------------: |
| qwen4exp 125B.A6B Q4\_K - Medium | 103.68 GiB | 176.94 B | CUDA | 999 | 1024 | 1 | graph | 1.00/1.75 | pp4096 | 380.67 ± 6.25 |
| qwen4exp 125B.A6B Q4\_K - Medium | 103.68 GiB | 176.94 B | CUDA | 999 | 1024 | 1 | graph | 1.00/1.75 | pp8192 | 360.75 ± 5.05 |
| qwen4exp 125B.A6B Q4\_K - Medium | 103.68 GiB | 176.94 B | CUDA | 999 | 1024 | 1 | graph | 1.00/1.75 | tg128 | 26.29 ± 0.08 |
build: 1a2a8604 (4878)
I would like to hear more abouth the flags, since I am just noob in this...
**thank you for your attention to this matter**document fetched 2026-09-20 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1wkwec5/i_turned_an_asymetric_pair_of_tesla_v100s_pcie