Tandem-vLLM 0.2.0: DeepSeek-V4.1-Flash now on 3090/4090-class, A6000, Ada and H100/H200 cards, and on two GPUs (a second card as an "expert helper")
retained text
Tandem-vLLM is an expert-offload build of vLLM 0.31 for DeepSeek-V4.1-Flash. 0.1.1 ran only on Blackwell. 0.2.0 adds Hopper, Ada and Ampere GPUs, and two-GPU setups. * **Engine:** [https://github.com/ewhacc/tandem](https://github.com/ewhacc/tandem) * **Model package:** [https://huggingface.co/ewhacc/DeepSeek-V4.1-Flash-1017-Tandem-VQ2](https://huggingface.co/ewhacc/DeepSeek-V4.1-Flash-1017-Tandem-VQ2) (293 GB) **How it works** * The routed experts live in host RAM as 2-bit vector codes (VQ, 2.1 bits/weight). A GPU cache holds the most-used ones. * Experts that miss the cache are computed by the CPU in parallel with the GPU. Nothing is skipped or substituted, so the output is exact for the package's weights. * Everything else is stock vLLM: OpenAI-compatible server, continuous batching, prefix caching, tool calling, image input, DSpark speculative decoding. **New in 0.2.0** * **GPUs:** RTX 30xx / A6000 (Ampere), RTX 40xx / 6000 Ada (Ada), H100 / H200 (Hopper), plus Blackwell as before. Each kernel is checked against a reference at boot. * **Two GPUs, two ways:** * **Tensor parallel** (`GPU=0,1`): best with a strong CPU. * **Helper GPU** (`HELPER_GPU=1`): the second card caches the hot experts the main card doesn't hold and computes them, so the CPU does less. Best with a smaller CPU or 24 GB cards. Needs no P2P or NVLink. **Speed:** one user, GSM8K, decode tok/s. The CPU matters as much as the GPU. |GPUs|CPU|one GPU|tensor parallel|helper GPU| |:-|:-|:-|:-|:-| |2× RTX PRO 6000 96 GB|EPYC 9534, 64 cores|75.5|**93.1**|| |2× RTX A6000 48 GB|EPYC 7F32, 8 cores|22.1|11.8|**26.7**| |2× 24 GB (A6000s limited to 24 GB)|EPYC 7F32, 8 cores|16.8|12.5|**21.7**| |one GPU|CPU|tok/s| |:-|:-|:-| |H200|Threadripper PRO 7965WX|60.7| |RTX 6000 Ada 48 GB|EPYC 7F32, 8 cores|25.5| * With 8 cores, tensor parallel is slower than one GPU because each GPU gets half of the CPU. The helper GPU keeps all of the CPU for one main card. * The 2× 24 GB row is the 3090/4090 memory shape, run on A6000s limited to 24 GB. Real 4090 numbers are coming. * A 24 GB card is the minimum main GPU. 16 GB cards don't fit as the main card. * At 32 streams, one RTX PRO 6000 serves 414 tok/s in total. **Quality vs the official FP4 checkpoint** (greedy, thinking low) |official|VQ + FP4 top 52|VQ only| |:-|:-|:-| |GPQA-diamond|81.3%|76.8%|73.2%| |GSM8K-200|192|194|192| |passkey 8k / 32k / 94k|9/9|9/9|9/9| The 2-bit experts cost GPQA accuracy; easier tasks and long-context retrieval don't show it. `FP4_TOP=52` keeps the 52 most-routed experts per layer in original FP4 and wins back about half of the loss. **What you need** * 300 GB of disk for the package, and 128 GB+ of RAM (256 GB to pin everything). * Linux, CUDA 13 toolkit, an AVX2 or AVX-512 CPU. One installer script, its own venv, no Docker. **Built on** [vLLM](https://github.com/vllm-project/vllm) and [vLLM-Moet](https://github.com/kacper-daftcode/vLLM-Moet). Tandem is a fork of vLLM-Moet's expert-offload engine, ported to vLLM 0.31 / DeepSeek-V4.1. Numbers from other boxes are welcome, especially real 3090s and 4090s. *P.S. For one user on a 16-32 GB card with 64-128 GB of RAM, our llama.cpp-based sister project* [*tandem.cpp*](https://github.com/ewhacc/tandem.cpp) *is faster (33 tok/s on a 5090 + 128 GB).*
document fetched 2026-11-14 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xvjoa7/tandemvllm_020_deepseekv41flash_now_on