← sources

vllm-project/vllm v0.34.0: v0.34.0

source: github (github:vllm-project/vllm)published 2026-11-16 23:22ingested 2026-11-17 02:34

extract

# v0.34.0 **1,061 commits from 412 contributors (120 new).** Thank you! ## TL;DR - **10 new models** (Mistral Large 4, MiniMax-M3.1-Flash, Nemotron-3.5-Transcribe, and more!) - **Extensible KV cache is on by default:** vLLM sizes the KV cache from memory actually free after warmup, so most models get a bigger cache and more throughput. - **Model Runner V1 is gone.** Model Runner V2 is now the only GPU runner. - **Big speedups** on Blackwell and MI355X: up to 1.95× decode (Step5 on GB200), 1.6× throughput (GLM-5.3-Flash), 1.24–1.40× faster long-prompt TTFT (GLM-5.3). - **Upgrade check:** default block size is now 64, ROCm 7.2 images are gone, and `huggingface_hub >= 2.0.0` is required. See [Before you upgrade](#before-you-upgrade). ## Before you upgrade > [!IMPORTANT] > These changes can break existing setups. | Change | What to do | PR | |---|---|---| | Model Runner V1 removed | Remove `VLLM_USE_V2_MODEL_RUNNER=0`; it now fails at startup. Port V1 logits processors to `vllm.v1.worker.gpu.sample.logits_processor`. | #63373 | | Default `--block-size` is 64 (was 16) | Pass `--block-size 16` to keep the old behavior. ROCm paged attention still uses 16. | #63231 | | Extensible KV cache on by default; `gpu_memory_utilization` defaults to 1.0 | To keep the old sizing, pass `--gpu-memory-utilization 0.9` or `--no-enable-extensible-kv-cache`. | #56492 | | ROCm 7.2 artifacts dropped | Use the ROCm 10.0 wheels and images. `-rocm72` is no longer published. | #61939 | | `huggingface_hub >= 2.0.0` required | Also requires `tokenizers >= 0.23.3` and `timm >= 1.0.30`. | #64779 | | QuTLASS / `fp_quant` removed | `fp_quant` checkpoints no longer load. | #65230 | | Marlin W4A8 removed | Use `--quantization humming` with `VLLM_HUMMING_INPUT_QUANT_CONFIG` instead of `VLLM_MARLIN_INPUT_DTYPE`. | #65542 | | Keye-VL and Keye-VL-1.5 removed | — | #64287 | All breaking changes, deprecations and new defaults are listed in [the full changelog](#full-changelog). ## New models | Model | Notes | PR | |---|---|---| | **Mistral Large 4** | 1T total / 52B active, FP8 and NVFP4. Runs on NVIDIA Blackwell/Hopper and AMD MI350. | #64534 | | MiniMax-M3.1-Flash | Blackwell NVFP4 and ROCm kernels, DSpark draft | #64111 | | NVIDIA Nemotron-3.5-Transcribe | 30B-A4B speech model: audio chat and `/v1/audio/transcriptions` | #64282 | | Ricursive Ripple-65B Preview | BF16 and NVFP4, `ripple` reasoning and tool parsers | #65745 | | Zyphra ZAYA1 | ZAYA1-8B, ZAYA1-74B-preview | #65302 | | Baidu Unlimited-OCR-2 | DASH sparse attention | #64016 | | Liquid AI LFM3, LFM3-VL, LFM3-Audio | | #65826, #65827, #65828 | | NVIDIA Canary-1B-v2 | ASR and X↔En translation | #64215 | | Ming3-ASR | | #63585 | | NVIDIA Parakeet TDT | Through the vllm-parakeet-plugin | #65005 | ## Highlights ### Extensible KV cache, on by default - On CUDA and ROCm, the KV cache is sized from memory actually free **after** CUDA graph capture and warmup, not from an estimate (#56492). - **Result:** a larger cache (e.g. MiniMax-M3 on H200: 433k → 672k tokens), so throughput is higher and TTFT lower at high concurrency. - **Trade-off:** per-token decode latency can rise on some models (TPOT +6–18% on some H200 models at concurrency 128). Opt out with `--gpu-memory-utilization 0.9` or `--no-enable-extensible-kv-cache`. - Hardened with fixes for allocator fragmentation, startup OOMs and KV connectors (#67296, #67485, #66731, #66523, #66197, #65137, #66605, #66674). ### Model Runner V2 only - Model Runner V1 is removed (#63373). - Dual-batch overlap now runs on MRV2, including with speculative decoding and DCP (#61464). ### Speculative decoding - **Suffix decoding** runs on MRV2 with a GPU speculator; Arctic Inference is no longer needed (#60339). - **Custom speculators:** declarative configs plus a `register_speculative_method` plugin API (#63205). - **CUDA graphs** for MRV2 draft-model speculators (#60989). - **Adaptive verification** works with XPress (#65175). - **Draft KV cache matches the target's dtype**, so an FP8 target gets an FP8 draft cache (#65198). ### Performance | Improvement | Hardware | Gain | PR | |---|---|---|---| | Step5 NVFP4 MoE on TRTLLM-Gen | 4×GB200 | up to **1.95×** decode | #65152 | | GLM-5.3-Flash DeepGEMM FP8 MegaMoE | NVIDIA | **1.6×** output throughput vs DEP4 | #60895 | | Muse Spark Open specialized decode (opt-in) | GB300 | **1.5×** single-stream decode | #64077 | | Token-sharded sparse-MLA prefill under TP (default) | GB200 | GLM-5.3 TTFT **1.24–1.40×** faster at 32K–128K | #63956 | | KV-balanced MLA decode for Kimi-K3 (opt-in) | GB300 | **1.34–1.59×** decode attention | #63638 | | Qwen3.5 on breakable CUDA graphs (no torch.compile) | NVIDIA | **3.0–8.8%** lower decode ITL | #51012 | | FlyDSL decode monokernel, Muse Spark Open DSpark | 8×MI355X | **+35–57%** interactivity at conc. 1–8 | #63440 | ### AMD ROCm - Release images and wheels are **ROCm 10.0 only** (#61939). - MoRI EPv2 via `--all2all-backend mori` (#61361). - Kimi-K3 specialized decode on MI355X, opt-in (#61435). - TRITON_ATTN is the default attention backend on RDNA3 and RDNA4 (#60418). ### Serving and operations - **Checkpoint/restore** (CRIU + `cuda-checkpoint`) through `/checkpoint/prepare`, `/checkpoint/status` and `/checkpoint/restore`, in both the Python and Rust frontends (#62570). - **Fault-tolerant DP+EP scale-down:** dead ranks are removed and their experts redistributed without a restart (#46370). - Best-effort **NUMA binding** on by default (#61258). - Opt-in **cache-aware admission** ordering (#54625). - Rust frontend: stateless **Responses API** (#63902) and **`n > 1`** (#65625). ## Install **Python wheels** | Platform | Install | |---|---| | CUDA 13.0 (PyPI) | `pip install vllm` | | CUDA 13.0 (uv) | `uv pip install vllm --torch-backend=auto` | | ROCm 10.0 | `pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.34.0/rocm100` | | XPU | `uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.34.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match` | <!-- TODO(reviewer): confirm the XPU install line still resolves now that vllm_xpu_kernels==0.1.17 comes from https://vllm-project.github.io/vllm-xpu-kernels/whl (#64633, #60609); the release index may need that extra index too. --> **Docker images** | Platform | Image | Short alias | |---|---|---| | CUDA 13.0 (default) | `vllm/vllm-openai:v0.34.0` | `vllm/vllm:v0.34.0` | | CUDA 13.0 + DeepEP over AWS EFA | `vllm/vllm-openai:v0.34.0-efa` | `vllm/vllm:v0.34.0-efa` | | ROCm 10.0 | `vllm/vllm-openai-rocm:v0.34.0` | `vllm/vllm:v0.34.0-rocm` | | CPU | `vllm/vllm-openai-cpu:v0.34.0` | `vllm/vllm:v0.34.0-cpu` | | XPU | `vllm/vllm-openai-xpu:v0.34.0` | `vllm/vllm:v0.34.0-xpu` | `latest` tags work the same way (e.g. `vllm/vllm:latest-rocm`). The default CUDA image now includes the AWS EFA userspace (libfabric, aws-ofi-nccl). The `-efa` image adds DeepEP built for EFA and routes NCCL over EFA by default (#65576). The **Assets** section below has the source tarball and the CUDA 13.0 (x86_64, arm64), CPU (x86_64, arm64, macOS) and XPU (x86_64) wheels. ## Full changelog <details> <summary><b>Model support</b></summary> - **Mistral Large 4** (#64534) runs on MRV2 with: - sparse MLA plus the DSA indexer, and cross-layer KV sharing; - DSpark, DCP/PCP, NIXL P/D and HiSparse. - **Other new models:** - MiniMax-M3.1-Flash (#64111), plus P/D across different prefill and decode TP (#64475). - Nemotron-3.5-Transcribe with MTP and DFlash2 (#64282, #64944). - **DeepSeek-V4 / V4.1** - Prefill gathers compressed KV once per KV-source group: 38 → 4 gathers per step (#59131). - Decoder replay graphs share the model graph pool: prefill graph memory 15.52 → 9.88 GiB per rank on GB200 (#63155). - `fused_inv_rope_fp8_quant` 3.5× faster, prefill 6% faster (#63540). - DSpark/MTP memory savings of about 512 MiB per GPU (#63996). - Engram host tables shared across DP ranks (#62359). - DeepGEMM sparse MQA logits on the FP8 indexer cache (#62011). - Packed multi-image vision encoding, 1.6–2.6× (#65030). - LoRA on the language model, compressor and `wo_a` layers (#57245, #64880). - DSpark with PCP (#61994); PCP with PP or DP (#63098). - Fixes: sparse prefill in FULL CUDA graphs (#59986), prefill memory budgeting (#63047), MTP embedding under PP (#65816), DeepSeek-V3.2 PP (#59961). - **GLM** - GLM-5.3-Flash: DeepGEMM FP8 MegaMoE (#60895), adaptive verification (#60229), row-strided QKV into FlashKDA (#61614). - GLM-5.3 FP8 MoE sorted-A TMA on H200, 6.0–7.6% E2E throughput (#64760). - FP8 KV cache for DSpark and DFlash drafts (#63841, #64612, #65553). - FP32 indexer MQA logits restored for GLM-5/GLM-5.3/HY-V4, with new `--attention-config.indexer_logits_dtype` (#64580). - Fused KDA convs fix stale weights after RL refit (#60991). - **Muse Spark** - Opt-in Muse Spark Open specialized decode on GB300 (#64077). - One RoPE cache shared across layers: 11.5 GiB per GPU saved, 14.5% more KV cache on 8×B200 (#65206). - GEMM and communication+norm fusions (#65438, #64638); hybrid DeepGEMM MegaMoE / TRT-LLM for NVFP4 (#64525). - CUDA routing kernel (#64553), MegaMoE JIT warmup (#63920), DSpark draft loading fix (#65125). - **MiniMax-M3 / M3.1** - Sequence parallelism and GEMM-RS for TP + MegaMoE, 14–20% lower P50 TTFT (#64659), also with PP (#65537). - Fused norm + MXFP8 quant, 1.39–1.60× (#64852). - Triton query-tiled sparse prefill on Hopper (#57420); SM90 split-K indexer (#64037). - No host sync in prefill metadata (#65883); NVFP4 MegaMoE warmup (#66227); fmha_sm100 indexer updates (#66228). - Fixes: M3.1 prefill workspace (#64856), sequence parallelism (#65496), encoder CUDA graphs (#65226). - **Kimi** - Opt-in `--attention-backend KIMI_K3_KV_BALANCED_MLA` for agentic workloads on Blackwell (#63638). - Kimi-K3 MegaMoE symmetric buffer 10.73 → 2.50 GiB per GPU at TP8 EP (#63541). - FlashInfer fused FP8 chunked-context K/V pack (#61527). - Fixes: KDA prefill OOM (#63989), NaNs via FlashKDA/FlashMLA pins (#64759), sleep mode (#64583, #66273). - Kimi K3.1 groundwork landed (#66030, #66028), but **K3.1 is not servable in this release**. - **Qwen** - Qwen3.5: breakable CUDA graphs instead of torch.compile (#51012); fused norm/gate + quant (#64509) and AR + norm + quant (#64723); GDN `in_proj` overlap (#64425, #64999). - GDN prefill state gather/scatter fused for Qwen3-Next/Qwen3.5 and Kimi KDA, 1.16–1.50× (#66064). - Qwen3.8-Flash-Next: PLE disk offload via `--engram-config '{"disk_offload": true}'` (27 GiB less on DGX Spark, #59958), PP (#60234), shared QSA top-k buffer (#65619), loading fixes (#62318, #65812, #64820, #60490). - Fixes: long-context int32 overflow (#63761), multi-layer MTP (#62111). - **Other models** - Step5 MegaMoE with PP (#65504) and KV block sizing (#65153). - Gemma 4: multimodal encoder DP (#60314), PP (#65989), FA4 FP8 MMA on SM90 (#60372). - Moved to the Transformers modeling backend: OLMoE (#63384), Gemma v1 and StableLM (#66393), Aria (#59767; temporarily about 11% less KV cache and 25% lower image throughput). - Matryoshka dimensions for EmbeddingGemma 2 (#64885, #60978); RADIO C-RADIO v2/v3/v4 (#58468). - Qwen3-ASR on the upstream Transformers config; no `--hf-overrides` needed for the ForceAligner (#66097, #63274). - Fixes: GPT-OSS MXFP4 LoRA (#65825), PLaMo3 GPTQ (#62836), Ling3 (#65088), DomynEdge (#66819). - **Multimodal** - Encoder CUDA graphs for Pixtral/Mistral3 (#56277), shared capture buffers (#65084), one replay per batch up to `encoder_cudagraph_max_token_budget` (#65227). - On-device input normalization for Nemotron VL (#57924), Gemma 3 (#64487), Gemma 4 (#65577). - `--mm-encoder-tp-mode data` for MiMo-V2.6 (#64179). - Fixes: FlashInfer ViT attention on GLM-4.1V/GLM-OCR/GLM-5.3-Flash (#65778), Qwen3-Omni and Qwen2.5-Omni video budgets (#66017, #66190), dots_ocr with TP (#64543), embedding-only startup (#65689), media cache eviction (#65363). </details> <details> <summary><b>Engine core</b></summary> - **Extensible KV cache** - On by default on CUDA and ROCm (#56492). - Headroom for measured allocator fragmentation, validated on Qwen3.5-397B-A17B-FP8 at TP8+EP on 8×H200 (#67296). - Fixes: sizing and startup OOMs (#67485, #66523, #66731, #66197, #65554), connectors that failed to start (#65225), up to 84% undersized caches on DGX Spark (#65413), `--max-model-len auto` (#65463). - Startup warning when an explicit `gpu_memory_utilization` withholds memory from it (#66704); docs on when to set it (#67071). - **Model Runner V2** - The only runner after MRV1's removal (#63373); DBO with spec decode and DCP (#61464). - Separate prefill (token) and decode (request) CUDA graph capture sizes (#63406). - Staged writes copied straight into UVA (#63633); idle sampler writes skipped, about 10–11% faster small-batch decode (#62748). - Sampler warmed up after graph capture to stop memory growth under load (#62560). - Lower peak memory for chunked prompt logprobs (#64841, #64914); Triton ReplaySSM (#65033). - NVIDIA Confidential Computing decode fixes (#59981). - **Speculative decoding** - Suffix decoding (#60339) and custom speculators (#63205) on MRV2; draft-model CUDA graphs (#60989). - XPress with adaptive verification (#65175); adaptive verification draft budgets on a Triton kernel (#63421). - Batch-sharded draft logits (#63554); DSpark sequential blocks (#63886). - DSpark sliding-window MLA draft layers (#63795) under DCP (#63796); fused Markov sampling, 42–50% less sampling time (#64476); `draft_lm_head_quantization` (#64368). - Draft KV cache dtype aligned with the target (#65198). - Trace replay with ngram/EAGLE3/MTP, 1.47–2.03× (#63275); 3× faster resample kernel (#65100). - Fixes: DFlash acceptance of 0 with FULL CUDA graphs (#64059), a resample race (#66048), acceptance-estimator cancellation (#65229), standalone draft prefill under FULL graphs (#65804), grammar-invalid drafts (#61557). - **Scheduler and engine overhead** - Prompt token ids kept as int32 buffers: 40× less per-request CPU on a 128K prompt (#62702). - Per-request id lists kept out of the cyclic GC (#65300); 45–50% less KV block allocation CPU (#65831). - New requests and aborts handled while waiting for model output (#65466), with a cheaper wait loop (#65773). - Opt-in cache-aware admission via `--cache-aware-admission-window` (#54625). - `--max-num-inflight-kv-loads` starts async KV loads while the batch is full (#65319), with cached reservations: 47.2 → 5.4 ms per schedule (#65285). - A multimodal item is no longer split by a prefix hit (#65271). - **Startup and compile** - No openai/fastapi imports in EngineCore and workers (#65423, #65589). - CUDA graph memory profile skipped when its estimate is disabled (#64622). - Device-neutral compile cache via `VLLM_COMPILE_ON_ONE_RANK`, on by default (#63247); all graph-affecting config fields hashed into compile cache keys (#65652). - `--shuffle-safetensors-files` for large EP startups (#42837). - `BasevLLMParameter`'s Python `__torch_function__` dropped: GLM-5.2-FP8 TTFT −6.1% on GB300 (#63785). - **Lifecycle** - Process checkpoint/restore over HTTP (#62570); opt-in multi-worker CUDA checkpoint sleep (#64186). - Best-effort NUMA binding by default; `--no-numa-bind` to disable (#61258). - Sleep mode with exact-size pinned backups, up to about 578 GiB less host memory per node on Mistral Large 4 TP8 (#64630); level 2 discards only parameters (#66010, #66165, #59994). - Fast Start weight-cache restarts for online FP8/MXFP8, ModelOpt CUTLASS FP8/NVFP4/block-FP8 and AWQ/GPTQ Marlin (#63672, #63754, #64119, #64182), with pluggable shared warmup-cache stores (#65168). - `WatermarkConfig.enforce` (#62966). - **RL and weight loading** - `VLLM_RELOAD_MODE=direct` writes checkpoint tensors straight into the live model (#64336, #64352, #65281). - Every load and update finishes through `finalize_model_weights` (#65914, #64357, #64692, #57678). - Load plans for weight-transfer engines (#65235); per-update NCCL source ranks for PP trainers (#62557). - Sparse weight patches validated in one pass, 40–48% shorter pause (#66021). - Offloaded KV tier survives pause/sleep (#57810); partial stacked weights rejected (#63752). - **LoRA** - MoE-LoRA shrink/expand kernels up to 64 tokens (#64653); embedding LoRA scaled by `embed_scale` (#62470). - Rejected: adapters on unwrapped modules (#63489), non-linear expert placement with EP (#62797). - Fixes: reloads with requests in flight (#64616), KV-sharing fast prefill (#61162), SEQ_CLS heads (#63286), fused MoE expand rounding (#63894), Mamba-1 (#64681). - **Batch invariance:** LFM2 ShortConv (#62367), FlashInfer FP8 bmm and W4A16 NVFP4 (#62192), GPTQ (#60479), FA sliding window (#65627), `aten::mm` out_dtype overloads (#60047). </details> <details> <summary><b>Large-scale serving</b></summary> - **Expert parallelism and fault tolerance** - DP+EP scale-down via `POST /v1/fault_tolerance/apply` (#46370); MoRI EPv2 (#61361). - Opt-in `symm_mem` EPLB weight-migration backend (#62009). - `--enable-ep-weight-filter` with EPLB: DeepSeek-R1 NVFP4 DEP8 reads 105 GiB per rank instead of 385 GiB (#64187). - `num_redundant_experts` defaults to the minimum valid value (#30614). - Skipped combine staging copy in `flashinfer_nvlink_one_sided` (#65498); DeepEP V2 padded rows (#54109). - **Context parallelism and sparse attention** - Token-sharded sparse-MLA prefill under TP (#63956); DCP on the FlashMLA sparse bf16 KV path (#61609). - MLA decode up-projected before the A2A merge with `--dcp-comm-backend a2a` (#64994); faster DCP sparse-indexer top-k (#65178, #64129). - Cross-layer KV cache sharing for MLA layers (#64095). - Fixes: PP rank derivation under PCP (#65539), FlashInfer MLA DCP decode (#59392). - **KV connectors** - NIXL: push mode via `transfer_mode` (#64230); `ucx_backend_params` (rocm_ipc on MI355X: 3.4 → 483.7 Gbps, #62905); 38% fewer hybrid SSM transfer bytes (#62317); block pairing across different P/D block sizes (#59746), with mismatched push handshakes rejected (#59747); same-host NVLink with the extensible KV cache (#65137). - Mooncake: selective leases (#63420), automatic `cache_prefix` (#65160), per-GPU RDMA devices (#65076), heterogeneous-TP hybrid sharing (#54307), sleep mode (#59934), lease renewal (#64193). - Lookahead block hashes in NIXL, Mooncake and offloading connectors (#64096); UMBP standalone mode (#61795, #61793). - Out-of-tree and LMCache MP connectors with the extensible KV cache (#66605, #66674). - **KV offload** - SimpleCPUOffload with one host buffer per rank and opt-in HugeTLB pages (#60130); no zero-filled pool (#64345). - KVCR v0.2.0 (#64224, #63889); async lookups published per request group (#57474); secondary tiers rejected on multi-node engines (#62930). - Hardened P2P tier (#63781, #64250, #64256, #64251, #64501, #63618, #64300, #63988) and correctness fixes (#64229, #63566, #65279, #65631, #64400, #63937, #65640). - **HiSparse** - Uses the extensible KV cache by default: 16% more GPU+host KV capacity (#65214). - Fused slot invalidation, up to 1.8% lower TPOT on GLM-5.3-NVFP4 (#60103); fewer hash probes (#61263); speculative-group prefetch (#63801). - Platform backend for hardware plugins (#62592); scheduler deadlock fix (#62979). - **EPD and distributed** - Metadata-only image and video inputs for GLM-5.3-Flash and DeepSeek-V4.1 (#64071, #62802). - Bitwise-identical 1-stage and 2-stage custom all-reduce (#57042); FlashInfer all-reduce gated on 16-byte rows (#63123). - Cross-node MNNVL groups kept off CUDA IPC all-reduce

retained text (truncated)

# v0.34.0

**1,061 commits from 412 contributors (120 new).** Thank you!

## TL;DR

- **10 new models** (Mistral Large 4, MiniMax-M3.1-Flash, Nemotron-3.5-Transcribe, and more!)
- **Extensible KV cache is on by default:** vLLM sizes the KV cache from memory actually free after warmup, so most models get a bigger cache and more throughput.
- **Model Runner V1 is gone.** Model Runner V2 is now the only GPU runner.
- **Big speedups** on Blackwell and MI355X: up to 1.95× decode (Step5 on GB200), 1.6× throughput (GLM-5.3-Flash), 1.24–1.40× faster long-prompt TTFT (GLM-5.3).
- **Upgrade check:** default block size is now 64, ROCm 7.2 images are gone, and `huggingface_hub >= 2.0.0` is required. See [Before you upgrade](#before-you-upgrade).

## Before you upgrade

> [!IMPORTANT]
> These changes can break existing setups.

| Change | What to do | PR |
|---|---|---|
| Model Runner V1 removed | Remove `VLLM_USE_V2_MODEL_RUNNER=0`; it now fails at startup. Port V1 logits processors to `vllm.v1.worker.gpu.sample.logits_processor`. | #63373 |
| Default `--block-size` is 64 (was 16) | Pass `--block-size 16` to keep the old behavior. ROCm paged attention still uses 16. | #63231 |
| Extensible KV cache on by default; `gpu_memory_utilization` defaults to 1.0 | To keep the old sizing, pass `--gpu-memory-utilization 0.9` or `--no-enable-extensible-kv-cache`. | #56492 |
| ROCm 7.2 artifacts dropped | Use the ROCm 10.0 wheels and images. `-rocm72` is no longer published. | #61939 |
| `huggingface_hub >= 2.0.0` required | Also requires `tokenizers >= 0.23.3` and `timm >= 1.0.30`. | #64779 |
| QuTLASS / `fp_quant` removed | `fp_quant` checkpoints no longer load. | #65230 |
| Marlin W4A8 removed | Use `--quantization humming` with `VLLM_HUMMING_INPUT_QUANT_CONFIG` instead of `VLLM_MARLIN_INPUT_DTYPE`. | #65542 |
| Keye-VL and Keye-VL-1.5 removed | — | #64287 |

All breaking changes, deprecations and new defaults are listed in [the full changelog](#full-changelog).

## New models

| Model | Notes | PR |
|---|---|---|
| **Mistral Large 4** | 1T total / 52B active, FP8 and NVFP4. Runs on NVIDIA Blackwell/Hopper and AMD MI350. | #64534 |
| MiniMax-M3.1-Flash | Blackwell NVFP4 and ROCm kernels, DSpark draft | #64111 |
| NVIDIA Nemotron-3.5-Transcribe | 30B-A4B speech model: audio chat and `/v1/audio/transcriptions` | #64282 |
| Ricursive Ripple-65B Preview | BF16 and NVFP4, `ripple` reasoning and tool parsers | #65745 |
| Zyphra ZAYA1 | ZAYA1-8B, ZAYA1-74B-preview | #65302 |
| Baidu Unlimited-OCR-2 | DASH sparse attention | #64016 |
| Liquid AI LFM3, LFM3-VL, LFM3-Audio | | #65826, #65827, #65828 |
| NVIDIA Canary-1B-v2 | ASR and X↔En translation | #64215 |
| Ming3-ASR | | #63585 |
| NVIDIA Parakeet TDT | Through the vllm-parakeet-plugin | #65005 |

## Highlights

### Extensible KV cache, on by default

- On CUDA and ROCm, the KV cache is sized from memory actually free **after** CUDA graph capture and warmup, not from an estimate (#56492).
- **Result:** a larger cache (e.g. MiniMax-M3 on H200: 433k → 672k tokens), so throughput is higher and TTFT lower at high concurrency.
- **Trade-off:** per-token decode latency can rise on some models (TPOT +6–18% on some H200 models at concurrency 128). Opt out with `--gpu-memory-utilization 0.9` or `--no-enable-extensible-kv-cache`.
- Hardened with fixes for allocator fragmentation, startup OOMs and KV connectors (#67296, #67485, #66731, #66523, #66197, #65137, #66605, #66674).

### Model Runner V2 only

- Model Runner V1 is removed (#63373).
- Dual-batch overlap now runs on MRV2, including with speculative decoding and DCP (#61464).

### Speculative decoding

- **Suffix decoding** runs on MRV2 with a GPU speculator; Arctic Inference is no longer needed (#60339).
- **Custom speculators:** declarative configs plus a `register_speculative_method` plugin API (#63205).
- **CUDA graphs** for MRV2 draft-model speculators (#60989).
- **Adaptive verification** works with XPress (#65175).
- **Draft KV cache matches the target's dtype**, so an FP8 target gets an FP8 draft cache (#65198).

### Performance

| Improvement | Hardware | Gain | PR |
|---|---|---|---|
| Step5 NVFP4 MoE on TRTLLM-Gen | 4×GB200 | up to **1.95×** decode | #65152 |
| GLM-5.3-Flash DeepGEMM FP8 MegaMoE | NVIDIA | **1.6×** output throughput vs DEP4 | #60895 |
| Muse Spark Open specialized decode (opt-in) | GB300 | **1.5×** single-stream decode | #64077 |
| Token-sharded sparse-MLA prefill under TP (default) | GB200 | GLM-5.3 TTFT **1.24–1.40×** faster at 32K–128K | #63956 |
| KV-balanced MLA decode for Kimi-K3 (opt-in) | GB300 | **1.34–1.59×** decode attention | #63638 |
| Qwen3.5 on breakable CUDA graphs (no torch.compile) | NVIDIA | **3.0–8.8%** lower decode ITL | #51012 |
| FlyDSL decode monokernel, Muse Spark Open DSpark | 8×MI355X | **+35–57%** interactivity at conc. 1–8 | #63440 |

### AMD ROCm

- Release images and wheels are **ROCm 10.0 only** (#61939).
- MoRI EPv2 via `--all2all-backend mori` (#61361).
- Kimi-K3 specialized decode on MI355X, opt-in (#61435).
- TRITON_ATTN is the default attention backend on RDNA3 and RDNA4 (#60418).

### Serving and operations

- **Checkpoint/restore** (CRIU + `cuda-checkpoint`) through `/checkpoint/prepare`, `/checkpoint/status` and `/checkpoint/restore`, in both the Python and Rust frontends (#62570).
- **Fault-tolerant DP+EP scale-down:** dead ranks are removed and their experts redistributed without a restart (#46370).
- Best-effort **NUMA binding** on by default (#61258).
- Opt-in **cache-aware admission** ordering (#54625).
- Rust frontend: stateless **Responses API** (#63902) and **`n > 1`** (#65625).

## Install

**Python wheels**

| Platform | Install |
|---|---|
| CUDA 13.0 (PyPI) | `pip install vllm` |
| CUDA 13.0 (uv) | `uv pip install vllm --torch-backend=auto` |
| ROCm 10.0 | `pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.34.0/rocm100` |
| XPU | `uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.34.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match` |

<!-- TODO(reviewer): confirm the XPU install line still resolves now that vllm_xpu_kernels==0.1.17 comes from https://vllm-project.github.io/vllm-xpu-kernels/whl (#64633, #60609); the release index may need that extra index too. -->

**Docker images**

| Platform | Image | Short alias |
|---|---|---|
| CUDA 13.0 (default) | `vllm/vllm-openai:v0.34.0` | `vllm/vllm:v0.34.0` |
| CUDA 13.0 + DeepEP over AWS EFA | `vllm/vllm-openai:v0.34.0-efa` | `vllm/vllm:v0.34.0-efa` |
| ROCm 10.0 | `vllm/vllm-openai-rocm:v0.34.0` | `vllm/vllm:v0.34.0-rocm` |
| CPU | `vllm/vllm-openai-cpu:v0.34.0` | `vllm/vllm:v0.34.0-cpu` |
| XPU | `vllm/vllm-openai-xpu:v0.34.0` | `vllm/vllm:v0.34.0-xpu` |

`latest` tags work the same way (e.g. `vllm/vllm:latest-rocm`). The default CUDA image now includes the AWS EFA userspace (libfabric, aws-ofi-nccl). The `-efa` image adds DeepEP built for EFA and routes NCCL over EFA by default (#65576).

The **Assets** section below has the source tarball and the CUDA 13.0 (x86_64, arm64), CPU (x86_64, arm64, macOS) and XPU (x86_64) wheels.

## Full changelog

<details>
<summary><b>Model support</b></summary>

- **Mistral Large 4** (#64534) runs on MRV2 with:
  - sparse MLA plus the DSA indexer, and cross-layer KV sharing;
  - DSpark, DCP/PCP, NIXL P/D and HiSparse.
- **Other new models:**
  - MiniMax-M3.1-Flash (#64111), plus P/D across different prefill and decode TP (#64475).
  - Nemotron-3.5-Transcribe with MTP and DFlash2 (#64282, #64944).
- **DeepSeek-V4 / V4.1**
  - Prefill gathers compressed KV once per KV-source group: 38 → 4 gathers per step (#59131).
  - Decoder replay graphs share the model graph pool: prefill graph memory 15.52 → 9.88 GiB per rank on GB200 (#63155).
  - `fused_inv_rope_fp8_quant` 3.5× faster, prefill 6% faster (#63540).
  - DSpark/MTP memory savings of about 512 MiB per GPU (#63996).
  - Engram host tables shared across DP ranks (#62359).
  - DeepGEMM sparse MQA logits on the FP8 indexer cache (#62011).
  - Packed multi-image vision encoding, 1.6–2.6× (#65030).
  - LoRA on the language model, compressor and `wo_a` layers (#57245, #64880).
  - DSpark with PCP (#61994); PCP with PP or DP (#63098).
  - Fixes: sparse prefill in FULL CUDA graphs (#59986), prefill memory budgeting (#63047), MTP embedding under PP (#65816), DeepSeek-V3.2 PP (#59961).
- **GLM**
  - GLM-5.3-Flash: DeepGEMM FP8 MegaMoE (#60895), adaptive verification (#60229), row-strided QKV into FlashKDA (#61614).
  - GLM-5.3 FP8 MoE sorted-A TMA on H200, 6.0–7.6% E2E throughput (#64760).
  - FP8 KV cache for DSpark and DFlash drafts (#63841, #64612, #65553).
  - FP32 indexer MQA logits restored for GLM-5/GLM-5.3/HY-V4, with new `--attention-config.indexer_logits_dtype` (#64580).
  - Fused KDA convs fix stale weights after RL refit (#60991).
- **Muse Spark**
  - Opt-in Muse Spark Open specialized decode on GB300 (#64077).
  - One RoPE cache shared across layers: 11.5 GiB per GPU saved, 14.5% more KV cache on 8×B200 (#65206).
  - GEMM and communication+norm fusions (#65438, #64638); hybrid DeepGEMM MegaMoE / TRT-LLM for NVFP4 (#64525).
  - CUDA routing kernel (#64553), MegaMoE JIT warmup (#63920), DSpark draft loading fix (#65125).
- **MiniMax-M3 / M3.1**
  - Sequence parallelism and GEMM-RS for TP + MegaMoE, 14–20% lower P50 TTFT (#64659), also with PP (#65537).
  - Fused norm + MXFP8 quant, 1.39–1.60× (#64852).
  - Triton query-tiled sparse prefill on Hopper (#57420); SM90 split-K indexer (#64037).
  - No host sync in prefill metadata (#65883); NVFP4 MegaMoE warmup (#66227); fmha_sm100 indexer updates (#66228).
  - Fixes: M3.1 prefill workspace (#64856), sequence parallelism (#65496), encoder CUDA graphs (#65226).
- **Kimi**
  - Opt-in `--attention-backend KIMI_K3_KV_BALANCED_MLA` for agentic workloads on Blackwell (#63638).
  - Kimi-K3 MegaMoE symmetric buffer 10.73 → 2.50 GiB per GPU at TP8 EP (#63541).
  - FlashInfer fused FP8 chunked-context K/V pack (#61527).
  - Fixes: KDA prefill OOM (#63989), NaNs via FlashKDA/FlashMLA pins (#64759), sleep mode (#64583, #66273).
  - Kimi K3.1 groundwork landed (#66030, #66028), but **K3.1 is not servable in this release**.
- **Qwen**
  - Qwen3.5: breakable CUDA graphs instead of torch.compile (#51012); fused norm/gate + quant (#64509) and AR + norm + quant (#64723); GDN `in_proj` overlap (#64425, #64999).
  - GDN prefill state gather/scatter fused for Qwen3-Next/Qwen3.5 and Kimi KDA, 1.16–1.50× (#66064).
  - Qwen3.8-Flash-Next: PLE disk offload via `--engram-config '{"disk_offload": true}'` (27 GiB less on DGX Spark, #59958), PP (#60234), shared QSA top-k buffer (#65619), loading fixes (#62318, #65812, #64820, #60490).
  - Fixes: long-context int32 overflow (#63761), multi-layer MTP (#62111).
- **Other models**
  - Step5 MegaMoE with PP (#65504) and KV block sizing (#65153).
  - Gemma 4: multimodal encoder DP (#60314), PP (#65989), FA4 FP8 MMA on SM90 (#60372).
  - Moved to the Transformers modeling backend: OLMoE (#63384), Gemma v1 and StableLM (#66393), Aria (#59767; temporarily about 11% less KV cache and 25% lower image throughput).
  - Matryoshka dimensions for EmbeddingGemma 2 (#64885, #60978); RADIO C-RADIO v2/v3/v4 (#58468).
  - Qwen3-ASR on the upstream Transformers config; no `--hf-overrides` needed for the ForceAligner (#66097, #63274).
  - Fixes: GPT-OSS MXFP4 LoRA (#65825), PLaMo3 GPTQ (#62836), Ling3 (#65088), DomynEdge (#66819).
- **Multimodal**
  - Encoder CUDA graphs for Pixtral/Mistral3 (#56277), shared capture buffers (#65084), one replay per batch up to `encoder_cudagraph_max_token_budget` (#65227).
  - On-device input normalization for Nemotron VL (#57924), Gemma 3 (#64487), Gemma 4 (#65577).
  - `--mm-encoder-tp-mode data` for MiMo-V2.6 (#64179).
  - Fixes: FlashInfer ViT attention on 

document fetched 2026-11-17 02:34