# 1Cat-vLLM vs Strata — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/da230a03773a65a4

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-15 17:49 ingested 2026-11-16 02:38

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xx1u9y/1catvllm_vs_strata/)

## retained text



```
Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.

Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:

## Single stream

Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.

| Prompt (tokens) | Prefill tok/s, Strata | Prefill tok/s, 1Cat | Decode tok/s, Strata | Decode tok/s, 1Cat | MTP acceptance, Strata / 1Cat |
|:--|--:|--:|--:|--:|:-:|
| 2,654 | 861 | **1,850** | 74.8 | **124.9** | 0.61 / 0.45 |
| 8,674 | 1,146 | **2,483** | 79.3 | **140.8** | 0.67 / 0.66 |
| 30,299 | 2,480 | **2,776** | 76.4 | **139.3** | 0.63 / 0.58 |
| 96,588 | **3,384** | 2,649 | 77.2 | **142.6** | 0.72 / 0.62 |

Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.

## The same 103K-token prompt three times

Same prompt, 3 times in a row, just different enough to not trigger prefill cache.

| Run | Decode tok/s, Strata | Decode tok/s, 1Cat | Prefill tok/s, Strata | Prefill tok/s, 1Cat | MTP acceptance, Strata / 1Cat |
|:-:|--:|--:|--:|--:|:-:|
| 1 | 70.8 | 99.9 | 3,398 | 2,736 | 0.65 / 0.60 |
| 2 | 78.9 | 138.4 | 3,340 | 2,752 | 0.71 / 0.60 |
| 3 | 77.8 | 123.5 | 3,348 | 2,751 | 0.69 / 0.57 |
| run 1 to 3 | +10% | +24% | - | - | - |

## Several requests at once

(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: `--batch-mtp` with `STRATA_PIPED_MTP=1`, which keeps the pipelined layer-split groups and drafts inside them.

| Requests | Prompt | Total decode tok/s, Strata / Strata batch-MTP / 1Cat | Per-stream decode tok/s, Strata / batch-MTP / 1Cat | Slowest first token (s), Strata / batch-MTP / 1Cat |
|:-:|--:|:-:|:-:|:-:|
| 1 | 2.6K | 75 / - / **125** | 74.8 / - / **124.9** | 3.1 / - / **1.4** |
| 1 | 8.7K | 79 / - / **141** | 79.3 / - / **140.8** | 7.6 / - / **3.5** |
| 2 | 2.6K | 88 / 105 / **176** | 39.4 / 46.0 / **83.2** | 6.4 / 6.5 / **2.7** |
| 2 | 8.7K | 96 / 116 / **189** | 37.9 / 44.6 / **77.9** | 15.4 / 15.5 / **7.3** |
| 4 | 2.6K | 153 / 170 / **225** | 29.1 / 31.5 / **48.5** | 13.4 / 13.6 / **5.8** |
| 4 | 8.7K | 170 / 186 / **234** | 25.4 / 26.9 / **41.1** | 31.0 / 31.4 / **15.3** |

## Versions and local changes

| - | Strata | 1Cat-vLLM |
|:--|:--|:--|
| Upstream repo | github.com/Niko1221/Strata | github.com/1CatAI/1Cat-vLLM |
| Commit benchmarked | `13eedf55` (2026-11-12, upstream HEAD on the test day; engine 0.1.59.1) | `b7baf7771` (2026-11-15, upstream HEAD on the test day; merge of PR #1521; vLLM 1.5.2.dev0) |
| Source patches | none | none |
| Build | from source for sm_70 (the released engines have no sm_70): CUDA 12.9.1, gcc-14 | from source for sm_70: CUDA 12.9.1, gcc-14, torch 2.10.0+cu128, NCCL 2.27.5 |
| Build flags | `-DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON`, ggml from llama.cpp at Strata's own pin, native ggml-cpu for Zen 3 | `TORCH_CUDA_ARCH_LIST=7.0` / `CMAKE_CUDA_ARCHITECTURES=70` |
| Parallelism | layer split over 4 GPUs (no tensor parallel for V100 in Strata) | tensor parallel 4 |
| MTP drafter | Strata's own pack of the BF16 MTP layer, experts in Q2_0 | the BF16 MTP layer as released |

TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.
```



document fetched 2026-11-16 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1xx1u9y/1catvllm_vs_strata
