# Tandem-vLLM 0.2.0: DeepSeek-V4.1-Flash now on 3090/4090-class, A6000, Ada and H100/H200 cards, and on two GPUs (a second card as an "expert helper") — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/8c57b8216bfbe4e5

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-13 21:03 ingested 2026-11-14 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xvjoa7/tandemvllm_020_deepseekv41flash_now_on/)

## retained text



```
Tandem-vLLM is an expert-offload build of vLLM 0.31 for DeepSeek-V4.1-Flash. 0.1.1 ran only on Blackwell. 0.2.0 adds Hopper, Ada and Ampere GPUs, and two-GPU setups.

* **Engine:** [https://github.com/ewhacc/tandem](https://github.com/ewhacc/tandem)
* **Model package:** [https://huggingface.co/ewhacc/DeepSeek-V4.1-Flash-1017-Tandem-VQ2](https://huggingface.co/ewhacc/DeepSeek-V4.1-Flash-1017-Tandem-VQ2) (293 GB)

**How it works**

* The routed experts live in host RAM as 2-bit vector codes (VQ, 2.1 bits/weight). A GPU cache holds the most-used ones.
* Experts that miss the cache are computed by the CPU in parallel with the GPU. Nothing is skipped or substituted, so the output is exact for the package's weights.
* Everything else is stock vLLM: OpenAI-compatible server, continuous batching, prefix caching, tool calling, image input, DSpark speculative decoding.

**New in 0.2.0**

* **GPUs:** RTX 30xx / A6000 (Ampere), RTX 40xx / 6000 Ada (Ada), H100 / H200 (Hopper), plus Blackwell as before. Each kernel is checked against a reference at boot.
* **Two GPUs, two ways:**
   * **Tensor parallel** (`GPU=0,1`): best with a strong CPU.
   * **Helper GPU** (`HELPER_GPU=1`): the second card caches the hot experts the main card doesn't hold and computes them, so the CPU does less. Best with a smaller CPU or 24 GB cards. Needs no P2P or NVLink.

**Speed:** one user, GSM8K, decode tok/s. The CPU matters as much as the GPU.

|GPUs|CPU|one GPU|tensor parallel|helper GPU|
|:-|:-|:-|:-|:-|
|2× RTX PRO 6000 96 GB|EPYC 9534, 64 cores|75.5|**93.1**||
|2× RTX A6000 48 GB|EPYC 7F32, 8 cores|22.1|11.8|**26.7**|
|2× 24 GB (A6000s limited to 24 GB)|EPYC 7F32, 8 cores|16.8|12.5|**21.7**|

|one GPU|CPU|tok/s|
|:-|:-|:-|
|H200|Threadripper PRO 7965WX|60.7|
|RTX 6000 Ada 48 GB|EPYC 7F32, 8 cores|25.5|

* With 8 cores, tensor parallel is slower than one GPU because each GPU gets half of the CPU. The helper GPU keeps all of the CPU for one main card.
* The 2× 24 GB row is the 3090/4090 memory shape, run on A6000s limited to 24 GB. Real 4090 numbers are coming.
* A 24 GB card is the minimum main GPU. 16 GB cards don't fit as the main card.
* At 32 streams, one RTX PRO 6000 serves 414 tok/s in total.

**Quality vs the official FP4 checkpoint** (greedy, thinking low)

|official|VQ + FP4 top 52|VQ only|
|:-|:-|:-|
|GPQA-diamond|81.3%|76.8%|73.2%|
|GSM8K-200|192|194|192|
|passkey 8k / 32k / 94k|9/9|9/9|9/9|

The 2-bit experts cost GPQA accuracy; easier tasks and long-context retrieval don't show it. `FP4_TOP=52` keeps the 52 most-routed experts per layer in original FP4 and wins back about half of the loss.

**What you need**

* 300 GB of disk for the package, and 128 GB+ of RAM (256 GB to pin everything).
* Linux, CUDA 13 toolkit, an AVX2 or AVX-512 CPU. One installer script, its own venv, no Docker.

**Built on** [vLLM](https://github.com/vllm-project/vllm) and [vLLM-Moet](https://github.com/kacper-daftcode/vLLM-Moet). Tandem is a fork of vLLM-Moet's expert-offload engine, ported to vLLM 0.31 / DeepSeek-V4.1.

Numbers from other boxes are welcome, especially real 3090s and 4090s.

*P.S. For one user on a 16-32 GB card with 64-128 GB of RAM, our llama.cpp-based sister project* [*tandem.cpp*](https://github.com/ewhacc/tandem.cpp) *is faster (33 tok/s on a 5090 + 128 GB).*
```



document fetched 2026-11-14 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xvjoa7/tandemvllm_020_deepseekv41flash_now_on
