← sources

How to get more performance from dual A100 80GB?

source: reddit (reddit:LocalLLaMA)published 2026-10-12 23:29ingested 2026-10-13 02:42

retained text

Dual A100X PCIe (link at Gen4 x8, weird DPU cards) in a 2U EPYC 7402P box, 256GB DDR4.

Proxmox with the GPU's passed through to debian 13, 610.43.02 drivers.

llama.cpp, qwen3.8 flash next UD-Q4\_K\_XL, Q8\_0 MTP head, draft max 2, mmproj enabled. 

`-sm layer -fa on --parallel 7 --ctx-size 1835008 -ub 2048 --cache-ram 32768`

(7 slots x 262k), q8\_0 K/V

  
Real world single user performance sits around  50-70tk/s. When bumping things up to 2-3 users/sessions, generation drops like a rock, 10-15tk/s. Seeking advice on how to better serve my team at work. We need models worth running and performance that doesn't have us waiting around.

I'm very partial towards MoE models, so no 27B. Should we swap over to vLLM, or run an instance of qwen3.6-35b-a3b on each card?

Use case is long session agentic 'coding' (more running linux commands, probing hardware devices). I'm frustrated and not sure how to squeeze more performance out of these. Thanks in advance!

document fetched 2026-10-13 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x4h9k4/how_to_get_more_performance_from_dual_a100_80gb