vllm-project/vllm v0.30.0: v0.30.0
extract
# v0.30.0 ## Highlights This release features 762 commits from 315 contributors (104 new)! * **New models**: DeepSeek-V4.1-Flash (#56214, #56228, #56208) with the whole KV stored in MXFP8 through the FlashMLA V4.1 record on SM100 (#56893), DeepGEMM Mega-mHC (#56962), and async Engram prefetch with Engram DP sharding (#56512); DeepSeek-V4-Flash-Vision-Exp (#54566), also on ROCm (#55107) and with LoRA (#55897); GLM-5.3-Flash (#53906) with EPLB (#55119); K2-Horizon (#55063); Cohere Compass (#54774); Bailing V3 VL (#55921); Nanbeige4.2 via the Transformers backend (#56071); and a DeepSeek-V4 CPU backend with AVX512/AMX sparse MLA, indexer, mHC and compressor kernels (#55355). * **Fast Start**: a persistent per-GPU weight-cache daemon holds post-quantized, TP-sharded weights in GPU memory so restarting engines map them over CUDA IPC with `--load-format ipc_cache` instead of reloading from disk (#54921), now covering FP4 checkpoints (#55465) and multi-node TP (#55468). * **Watermarking**: Gumbel-max watermarked generation and detection with a keyed PRF, per-request opt-out and an example detection endpoint (#54053); dual-key Gumbel-max makes it compatible with speculative decoding (#56122); the Rust frontend forwards the per-request controls (#56338). * **HiSparse**: a host-resident tier for sparse-MLA decode that spills KV pages to pinned host memory under GPU pressure and serves top-k misses from a per-request GPU hot buffer, enabled through `HiSparseConnector` (#53781), with Prometheus counters (#56061), a host cache shared across TP ranks (#56629), and the attention config inferred from the connector (#57041). * **Model Runner V2**: dual-batch overlap in eager mode (#50945) and with FULL CUDA graphs for microbatched steps (#51700); MTP (#46994) and EAGLE3/DFlash/DSpark (#50514) speculative decoding under pipeline parallelism; adaptive verification for every draft-model speculator through an online acceptance estimator (#52228); gc frozen during graph capture, cutting capture from 12s to 2s and engine init from 28.9s to 8.2s on H200 (#54646); `--return-sampling-mask` compacted on GPU, fixing an about 2x RL step-time regression (#54901). * **Qwen3.8-Flash-Next performance**: separate prefill and decode QSA indexer kernels (#54513), fused PLE kernels (#54517), FP8 indexer cache (#54890), padded-index skipping in sparse GQA (#54873), fused PLE residual and QSA output gate (#55309), UVA PLE offload and Engram tensor parallelism via `--engram-config` (#54371), and torch.compile removed from the NVIDIA implementation so FP8 fits on a single GB300 (#55272). * **Kimi K3 performance**: native CUDA AttnRes default on SM100 (#54261), KDA mixed-batch gather/scatter removed (5.2-7.7% E2E throughput, #56159), grouped FP8 MLA cache insertion (4-6x kernel speedup at small batch, #55356), DSV3 low-latency GEMM on strided tensors (12-81% kernel speedup, #54565), overlapped TP8 KDA projections (#54697), FlashInfer KDA kernels (#55364), internal prefix checkpoints with partial prefix caching and speculative decoding (#53614), and symmetric DCP disaggregation for hybrid Mamba models (#55531). * **Large scale serving**: PCP+DCP on sparse-MLA models (#56157), PCP with single-module MTP and replicated DSpark (#56107) and decode-only FULL CUDA graphs (#53867), Elastic EP reusing CUDA graphs across reconfiguration (#54985), an opt-in FlashInfer PCIe IPC all-reduce for NVLink-less boxes (#53576), DeepEP v2 async finalize overlapping shared experts with combine (#52781), Mooncake Store heterogeneous TP sharing (#53129), a KVCR secondary-tier adapter (#53624), and encoder-cache sharing over NIXL (#47941) and Mooncake (#41567). * **Quantization**: targeted online quantization through `quantization_config.targets` (#51285) and on partially pre-quantized checkpoints from any quant method (#51392), W4A16 DSA with the `nvfp4_fp8_ds_mla` KV cache (#51724), FlashInfer CuTeDSL NVFP4 W4A16 default over Marlin on SM100/103 (#53014), NVFP4 in the torch linear backend (#53319), per-quantization linear backend overrides (#51204), AutoRound 2/3/5/6/7-bit on CUDA (#52890), and DeepSelect top-k for the DSA sparse indexer (#56464). * **Breaking changes**: scale-out endpoints are opt-in on plain `vllm serve` via `--enable-scale-out`, replacing `VLLM_ENABLE_SCALE_OUT_ENDPOINTS` (#54579, #55176); GPTQ activation ordering (`g_idx`) removed (#54809); items deprecated for 0.29 removed, including the `VLLM_PREFIX_CACHE_RETENTION_INTERVAL` and `VLLM_MM_HASHER_ALGORITHM` env vars (#55353); the `all` Mamba cache mode deprecated (#55041); `python -m vllm.entrypoints.grpc_server` deprecated in favor of `vllm serve --grpc` (#56746); YaRN aligned with Transformers so vendor YaRN aliases no longer re-scale `max_model_len` (#56446). ## Release Artifacts ### Python Wheels | Platform | Install | |---|---| | PyPI (CUDA 13.0) | `pip install vllm` | | PyPI (CUDA 13.0, uv) | `uv pip install vllm --torch-backend=auto` | | ROCm | `pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723` | | XPU | `uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.30.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match` | ### Docker Images | Platform | Docker Image | |---|---| | CUDA 13.0 (Default) | `docker pull vllm/vllm-openai:v0.30.0` | | CUDA 12.9 | `docker pull vllm/vllm-openai:v0.30.0-cu129` | | CUDA 13.0 + Ubuntu 24.04 | `docker pull vllm/vllm-openai:v0.30.0-ubuntu2404` | | CUDA 12.9 + Ubuntu 24.04 | `docker pull vllm/vllm-openai:v0.30.0-cu129-ubuntu2404` | | ROCm | `docker pull vllm/vllm-openai-rocm:v0.30.0` | | CPU | `docker pull vllm/vllm-openai-cpu:v0.30.0` | | XPU | `docker pull vllm/vllm-openai-xpu:v0.30.0` | ### Other Artifacts Pre-built release artifacts are available in the **Assets** section at the bottom of this page, including: - Source distribution tarball - CUDA 12.9 Python wheels for x86_64 and arm64 - CUDA 13.0 Python wheels for x86_64 and arm64 - CPU Python wheels for x86_64, arm64, and macOS - XPU Python wheel for x86_64 ## Model Support * **New models**: DeepSeek-V4.1-Flash (#56214, #56228, #56208), DeepSeek-V4-Flash-Vision-Exp (#54566) with ROCm support (#55107) and LoRA (#55897), GLM-5.3-Flash (#53906), K2-Horizon with its reasoning and tool parsers (#55063), Cohere Compass (#54774, #55949), Bailing V3 VL with MTP (#55921), Nanbeige4.2 via the Transformers backend (#56071), and a DeepSeek-V4 CPU backend (#55355). * **DeepSeek-V4.1-Flash**: whole KV in MXFP8 through the FlashMLA V4.1 record on SM100 (#56893), Mega-mHC from the DeepGEMM fork (#56962), mHC post block folded into the delayed pre projection (#56633), Triton-fused input metadata preparation (#56562), async prefetch for CPU-offloaded Engram lookups and Engram DP sharding (#56512), DSpark draft states collapsed before the SP all-gather (#56903), DSpark drafter no longer inheriting uninitialized EPLB state (#56387), XGrammar V4.1 schema constraints for strict tool parameters (#56408), Responses text parts (#56299), and image sentinel padding scoped to the Vision-Exp checkpoint (#56554). * **DeepSeek V4**: plain RoPE without YaRN on sparse SWA layers (#54815), `--kv-cache-dtype auto` resolving to `fp8_ds_mla` with FlashMLA (#45091), prefill sparse index workspace sentinel seeded (#55299), dequant gather grid sized by rows (#55061), optional Q-norm in the fused MLA epilogue and group_size=32 packed FP8 quant (#56215), sparse settings read from the text config for composite models (#56160), MegaMoE startup without EP fixed (#55914), and a Triton iHC pre/post fallback for HY V4 (#55059). * **GLM-5.3-Flash**: EPLB support (#55119), FlashKDA for KDA chunked prefill (1.7-3.8x faster than the Triton chunk path, #55737), dense and masked-MHA sparse prefill for the NoPE (256, 0, 256) layout (#55738), decode hot-path cleanups removing per-layer copies and a duplicate router GEMM (#55736), and the `glm5next` NVIDIA subtree now packaged in wheels (#55214). * **Qwen3.8-Flash-Next**: separate prefill and decode QSA indexer kernels with `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` (#54513), fused PLE kernels and merged K/V projections (#54517), sparse GQA skipping padded indices (#54873), FP8 indexer cache (#54890), fused PLE residual and QSA output gate (#55309), UVA PLE offload and Engram tensor parallelism via `--engram-config` (#54371), torch.compile removed from the NVIDIA path (#55272), compact indexer logits workspace (#54915), HC combine-norm reused for MTP input (#54687), PLE MTP metadata transfers (#55054), Hopper LL-GEMM tuning table (#54560), B200 FP8 MoE tuning (#55890), and fixes for FP8 PLE weight scales (#54722, #54882) and fused PLE conv strides (#55375). * **Kimi K3**: native CUDA AttnRes default on SM100 (#54261), ll_bf16 router weight prefetch (#53524), overlapped TP8 KDA projections (#54697), strided DSV3 GEMM (#54565), MLA decode epilogue latency (#54896), NVFP4 in_proj alignment routing to DeepGEMM (#55242), mixed-batch KDA without gather/scatter (#56159), grouped FP8 MLA cache insertion (#55356), FlashInfer KDA kernels (#55364), FlashKDA bf16 checkpoint state (#56485) and a stability bump (#54859), SiTU on the FlashInfer CuTeDSL MoE backend (#54606), internal prefix checkpoints with partial prefix caching and spec decoding (#53614), and fixes for RecoverSSM startup (#54634, #55774), GDN metadata (#54636), mixed-batch masks (#54781), checkpoint indexing (#55747, #56794), KDA illegal memory access (#55924), interleaved weight streams (#53379), NVFP4 MoE conversion OOM (#55407), and KDA projection overlap on Hopper (#55426). * **Transformers backend**: attention found with a fuser and vLLM's layer attached to it (#54941), generic merged-column linear fusion (#55301), QKV fuser for Gemma4 (#55690), all vocab embeddings replaced including Gemma 3n/4 per-layer tables (#54760, #55083), and attention sinks applied (#52156). * **Speculative decoding for models**: EAGLE3 for Sarvam MLA (#53052), Nemotron-H MTP with a separate, possibly quantized lm_head (#54574), NVFP4 DSpark gathered top-k projection for quantized Markov heads (#55713), Qwen3.5/3.6 multimodal MTP resolving `n_predict` from the text config (#55369), Qwen3 DSpark padded-vocab drafts (#55133), and GLM-OCR MTP weight loading (#49869) and position masking under CUDA graphs (#56447). * **Multimodal**: torchcodec audio decoding and a selectable `audio_backend` in `--media-io-kwargs` (#51826) with automatic decoding kept soundfile-first (#55642), torchaudio as the default resampler (#52598), `media_io_kwargs` in multimodal hashes (#54241), cache hash kwargs scoped by modality (#54918), empty video URLs with multimodal UUIDs (#54220), cached audio inputs with UUIDs (#56310), encoder CUDA graphs for MiniCPM-V 2.5/2.6/4.0 (#42785), Voxtral Realtime with `FULL_DECODE_ONLY` graphs (#51167), Triton/FlashInfer composite attention for multimodal prefixes (#56305), pruned sliding-window tiles for Gemma 4 multimodal prompts (up to 3-4x E2E, #53147), SDPA for BLIP-2 Q-Former (#55285), baddbmm Conformer scores (#55062), fused DeepEncoder relative bias (#55629), Gemma3n sparse GELU Triton kernel (#48498), and no duplicate text embedding in Qwen2.5-Omni (#55415). * **Correctness**: Gemma 4 / Gemma 3n fine-tuned checkpoints without KV projections on shared layers (#54917, #55864), Gemma4 video preprocessing without image kwargs (#56652) and unified audio profiling (#56721), MiniCPM-V 4.6 loading (#49417) and video embeds (#54633), InternVL video parser (#55779), Nano-Nemotron precomputed embeddings (#56398), OpenPangu multimodal merge (#55941) and sleep mode (#53696), Qwen3-VL and Cosmos3-Edge under CPU and PP (#53699, #43272), Qwen VL processors without a tokenizer (#54886), Molmo2 fps guard (#55893), CohereASR audio-token estimate (#53829), double BOS in `LLM.chat()` for multimodal models (#55288), decoded frame lists parsed as one video (#55326), swapped H/W in dummy video profiling (#56385), base64 video validation (#54323), Sentence Transformers 5.4+ configs (#54533), ColQwen3.5 pooler projector (#54847), torch.compile enabled for StableLM (#54969) and Sarvam MLA (#55817), and Nemotron latent-MoE skipping a redundant all-reduce at TP>1 (about 13% decode win, #52301). * **LoRA**: rsLoRA scaling in MoE expert packing (#55548), a log line when an adapter applies no weights (#55310), and `--lora-modules name=path` with `=` in the path (#53353). ## Engine Core * **Model Runner V2**: DBO in eager mode (#50945) and with FULL CUDA graphs (#51700), MTP (#46994) and EAGLE3-style drafts (#50514) under pipeline parallelism, adaptive verification for all draft-model speculators (#52228), gc frozen during capture (#54646), GPU-compacted sampling masks (#54901), UVA-backed `apply_write` (#55819), fast-prefill support (#56145), DP sync skipped for all speculator uniform decodes (#54856), prompt embeddings in decoder pooling (#56639), pooling post-processing on non-final PP ranks (#56666), kernels warmed before graph capture (#55341), `CUDAGraphStat` (#52358), Proton CUDA graph attribution (#51084), platforms without pinned memory such as WSL (#56908), draft `moe_backend` (#54788) and `attention_backend` (#54826) overrides honored, DCP metadata initialized after batch partitioning (#55212), and a `record_stream` fix in the PP draft broadcast (#55745). The MRV2-specific Triton JIT warmup migration was reverted (#56654). * **Speculative decoding**: opt-in `disable_eagle_block_drop` (#53388), adaptive verification on `FLASHINFER_MLA_SPARSE_DSV4` (#52724) and deferred until after kernel warmup (#55455), full-vocab EAGLE3 drafters skipping vocab remapping (#53458), DSpark preserving the target DCP config (#55472), DFlash drafters dropping FlashAttention's AOT schedule (#54374), DP token padding excluded from draft metadata (#55458, #56181), fastsafetensors deadlock avoided for PP drafts (#54416), Mamba state cached at the EAGLE resume position (#53945) and preserved for padded prompt tails (#55178), resumed requests padded in P/D (#55126), and warmup device selection (#55245). * **Prefix caching and scheduling**: both replay boundaries retained so an EAGLE resend of a block-aligned prompt still hits (#54713), zero-progress preemption cascades stopped for deferred KV frees (#49675), SWA layers taking the primary block size to avoid inflating the KV block LCM (#53007), align-mode Mamba states retired across null gaps (#55450), the `all` Mamba cache mode deprecated with MRV1 fallback (#55041), and `vllm:request_num_preemptions` histogram (#49984). * **Kernels**: FlashInfer ReplaySSM backend for Mamba2 (#52506), B12X causal paged attention for SM120/SM121 via `--attention-backend B12X_ATTN` (#52017), FlashInfer XQA decode on SM90 (#50439), DeepSelect top-k with `--sparse-indexer-topk-backend` (#56464), BF16x3 router GEMM default on SM100 for FP32-router models (#55899), split-row Triton top-p for small batches (#54651), manual `ActivationQuantFusionPass` producer for Llama (#51415), fused add-RMSNorm + NVFP4 (#51925), prefix-prefill tiling independent of KV page size (#54194), FlashInfer GDN prefill on SM12x (#55715), fused GDN MTP decode for SM110 (#53835), masked MoE activation for padded layouts (#51217), cooperative `batched_moe_align_block_size` (#53280), PDL in `fusedQKNormRopeKernel` (#55755) and weight prefetch in fused q/kv RMSNorm (#55020), XD-RoPE folded into M-RoPE (#56078), and the FlashAttention pin synced with upstream (#54819). * **Startup and warmup**: renderer warmup overlapped with engine core init (#54557), FlashInfer autotune no longer rerun on each source change (#54794), Qwen3.5/Qwen3-Next Triton kernels (#54797) and GDN gated RMSNorm (#54251) warmed at startup, Gemma 4 de-JITified (#55768), DeepSeek V4 and shared attention, sampling, DFlash and mHC kernels migrated to the JIT warmup registry (#50175, #53564, #53565, #50176, #53566, #56323, #50178) with parallel mHC pre-norm compilation (#56683), FlashInfer deferred MoE decode kernels autotuned before capture (#55377), and supplemental BF16 autotuning isolated (#57285). * **Fast Start**: weight-cache daemon with `--load-format ipc_cache` (#54921), FP4 checkpoints (#55465), multi-node TP (#55468), GPU UUID socket folders (#56669), per-client IPC tensor export so a copy-mode client cannot release zero-copy weights (#56472), and a clear error on non-CUDA/ROCm platforms (#56010). * **Watermarking**: Gumbel-max watermarked generation and detection (#54053) and dual-key Gumbel-max for speculative decoding with `allow_target_only_watermarking` and context deduplication (#56122). * **Sleep and RL**: NCCL communicator memory released in sleep mode (#51485), DP state synced on the first step of a wave so pause and sleep stop burning 32 dummy forwards per rank (#52957), pre-shared `ncclUniqueId` rendezvous for torch-free trainers (#53784), stale HPC QK-norm weights after refit fixed (#56706), and `session_id` on GPU `BlockStored` events (#51381). * **Robustness**: attention implementations must explicitly declare DCP support (#55780), unavailable piecewise CUDA graphs raise instead of producing garbled output (#54782), non-compiled models fall back to `FULL_DECODE_ONLY` graphs (#55095), MRV1 breakable graphs scoped to the piecewise path (#56312), sync checks cover paged async copies (#53491) and torch.compile (#56904), more H2D copies pinned (#54660, #55202, #56138), GDN capture metadata built without a device sync (#55404), queued work carried over when materializing the dedicated stream (#56382), FlexAttention recompiles avoided (#55888), persistent top-k falling back on low-shared-memory GPUs (#54110), KV page size scaled for hidden-state extraction with TP (#50894), out-of-bounds access in SM90 sparse MLA mixed batches (#56969), sparse-MLA piecewise capture crash (#56825), TRT-LLM ragged prefill regression (#55499), MoE fused sum int32 overflow (#50220), odd-row per-token-group quant cliff (#56478), dummy-load re-initialization (#56682, #56688), `warning_once` retaining the LLM instance (#54098), DSpark cache-group capability under `python -O` (#55234), sparse-MLA DCP stabilized for GLM PCP configurations (#55879), DeepGEMM FP8 warmup coverage (#56452), Triton autotune inspection skipped without Triton (#56676), `mm_device_do_normalize` under encoder CUDA graphs (#55370), VidCom2 frame budgets read once (#55331), the CUDA profiler start fix (#55237), FlashInfer all-reduce workspace failures logged as errors (#55127), and backend-neutral KV cache and MFU log lines (#55650). ## Hardware & Performance * **NVIDIA**: FlashInfer PCIe IPC all-reduce (#53576), DeepGEMM pinned to the vLLM fork 2.8.0 with SM120 and SM90 paged-MQA ports (#56876), public CUDA 13.4 Rubin build path (#54640, #56545), SM12x blockwise FP8 CTA raster swizzle for GB10/DGX Spark (#55180), W4A4 NVFP4 preferred on SM120/121 (#55170), H20 block-FP8 MoE tuning (+21%, #54668), an A100 PCIe fused-MoE config for Qwen3.5-122B TP2 (#55511), ReLU2 fused with static FP8 quant for Nemotron-H (#53793), and NVFP4 KV cache FMHA sped up (#55031). * **AMD ROCm**: TheRock base bumped to ROCm 10.0 (#55246) with nightly TheRock images (#55014), AITER 0.1.21.post2 (#52826, #55968), AITER custom AG/RS on the DP group (about 3% TPOT, #48247), multi-stream performance and rocprofiler fixes (#55099, #56098), bpreshuffled blockscaled FP8 GEMM (+4-8% QPS on DSv3, #51692), low-M FP32 router GEMM on gfx950 (#54845), DeepSeek V4 FHMoE shared-expert fusion (#53161), CSA multi-stream overlap (#51794), AITER OPUS for large sparse prefill (#54855), DSv4.1 mHC through AITER (#56503, #56513), MXFP8 dequant-once (#56560), strided DSA decode mask (#56628) and K=512 top-k tuning (#56743), MiniMax-M3 indexer and top-k work (#54682, #52664, #55235, #56170) with fused allreduce+GemmaRMSNorm (#54787), Kimi-K3 fused KDA prefill relanded (#54038) and fused MLA norms (#52494), GLM-5.2 kept on MRV1 (#53155), GLM-5.3-Flash MTP through ragged sparse MLA (#5
retained text (truncated)
# v0.30.0 ## Highlights This release features 762 commits from 315 contributors (104 new)! * **New models**: DeepSeek-V4.1-Flash (#56214, #56228, #56208) with the whole KV stored in MXFP8 through the FlashMLA V4.1 record on SM100 (#56893), DeepGEMM Mega-mHC (#56962), and async Engram prefetch with Engram DP sharding (#56512); DeepSeek-V4-Flash-Vision-Exp (#54566), also on ROCm (#55107) and with LoRA (#55897); GLM-5.3-Flash (#53906) with EPLB (#55119); K2-Horizon (#55063); Cohere Compass (#54774); Bailing V3 VL (#55921); Nanbeige4.2 via the Transformers backend (#56071); and a DeepSeek-V4 CPU backend with AVX512/AMX sparse MLA, indexer, mHC and compressor kernels (#55355). * **Fast Start**: a persistent per-GPU weight-cache daemon holds post-quantized, TP-sharded weights in GPU memory so restarting engines map them over CUDA IPC with `--load-format ipc_cache` instead of reloading from disk (#54921), now covering FP4 checkpoints (#55465) and multi-node TP (#55468). * **Watermarking**: Gumbel-max watermarked generation and detection with a keyed PRF, per-request opt-out and an example detection endpoint (#54053); dual-key Gumbel-max makes it compatible with speculative decoding (#56122); the Rust frontend forwards the per-request controls (#56338). * **HiSparse**: a host-resident tier for sparse-MLA decode that spills KV pages to pinned host memory under GPU pressure and serves top-k misses from a per-request GPU hot buffer, enabled through `HiSparseConnector` (#53781), with Prometheus counters (#56061), a host cache shared across TP ranks (#56629), and the attention config inferred from the connector (#57041). * **Model Runner V2**: dual-batch overlap in eager mode (#50945) and with FULL CUDA graphs for microbatched steps (#51700); MTP (#46994) and EAGLE3/DFlash/DSpark (#50514) speculative decoding under pipeline parallelism; adaptive verification for every draft-model speculator through an online acceptance estimator (#52228); gc frozen during graph capture, cutting capture from 12s to 2s and engine init from 28.9s to 8.2s on H200 (#54646); `--return-sampling-mask` compacted on GPU, fixing an about 2x RL step-time regression (#54901). * **Qwen3.8-Flash-Next performance**: separate prefill and decode QSA indexer kernels (#54513), fused PLE kernels (#54517), FP8 indexer cache (#54890), padded-index skipping in sparse GQA (#54873), fused PLE residual and QSA output gate (#55309), UVA PLE offload and Engram tensor parallelism via `--engram-config` (#54371), and torch.compile removed from the NVIDIA implementation so FP8 fits on a single GB300 (#55272). * **Kimi K3 performance**: native CUDA AttnRes default on SM100 (#54261), KDA mixed-batch gather/scatter removed (5.2-7.7% E2E throughput, #56159), grouped FP8 MLA cache insertion (4-6x kernel speedup at small batch, #55356), DSV3 low-latency GEMM on strided tensors (12-81% kernel speedup, #54565), overlapped TP8 KDA projections (#54697), FlashInfer KDA kernels (#55364), internal prefix checkpoints with partial prefix caching and speculative decoding (#53614), and symmetric DCP disaggregation for hybrid Mamba models (#55531). * **Large scale serving**: PCP+DCP on sparse-MLA models (#56157), PCP with single-module MTP and replicated DSpark (#56107) and decode-only FULL CUDA graphs (#53867), Elastic EP reusing CUDA graphs across reconfiguration (#54985), an opt-in FlashInfer PCIe IPC all-reduce for NVLink-less boxes (#53576), DeepEP v2 async finalize overlapping shared experts with combine (#52781), Mooncake Store heterogeneous TP sharing (#53129), a KVCR secondary-tier adapter (#53624), and encoder-cache sharing over NIXL (#47941) and Mooncake (#41567). * **Quantization**: targeted online quantization through `quantization_config.targets` (#51285) and on partially pre-quantized checkpoints from any quant method (#51392), W4A16 DSA with the `nvfp4_fp8_ds_mla` KV cache (#51724), FlashInfer CuTeDSL NVFP4 W4A16 default over Marlin on SM100/103 (#53014), NVFP4 in the torch linear backend (#53319), per-quantization linear backend overrides (#51204), AutoRound 2/3/5/6/7-bit on CUDA (#52890), and DeepSelect top-k for the DSA sparse indexer (#56464). * **Breaking changes**: scale-out endpoints are opt-in on plain `vllm serve` via `--enable-scale-out`, replacing `VLLM_ENABLE_SCALE_OUT_ENDPOINTS` (#54579, #55176); GPTQ activation ordering (`g_idx`) removed (#54809); items deprecated for 0.29 removed, including the `VLLM_PREFIX_CACHE_RETENTION_INTERVAL` and `VLLM_MM_HASHER_ALGORITHM` env vars (#55353); the `all` Mamba cache mode deprecated (#55041); `python -m vllm.entrypoints.grpc_server` deprecated in favor of `vllm serve --grpc` (#56746); YaRN aligned with Transformers so vendor YaRN aliases no longer re-scale `max_model_len` (#56446). ## Release Artifacts ### Python Wheels | Platform | Install | |---|---| | PyPI (CUDA 13.0) | `pip install vllm` | | PyPI (CUDA 13.0, uv) | `uv pip install vllm --torch-backend=auto` | | ROCm | `pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723` | | XPU | `uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.30.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match` | ### Docker Images | Platform | Docker Image | |---|---| | CUDA 13.0 (Default) | `docker pull vllm/vllm-openai:v0.30.0` | | CUDA 12.9 | `docker pull vllm/vllm-openai:v0.30.0-cu129` | | CUDA 13.0 + Ubuntu 24.04 | `docker pull vllm/vllm-openai:v0.30.0-ubuntu2404` | | CUDA 12.9 + Ubuntu 24.04 | `docker pull vllm/vllm-openai:v0.30.0-cu129-ubuntu2404` | | ROCm | `docker pull vllm/vllm-openai-rocm:v0.30.0` | | CPU | `docker pull vllm/vllm-openai-cpu:v0.30.0` | | XPU | `docker pull vllm/vllm-openai-xpu:v0.30.0` | ### Other Artifacts Pre-built release artifacts are available in the **Assets** section at the bottom of this page, including: - Source distribution tarball - CUDA 12.9 Python wheels for x86_64 and arm64 - CUDA 13.0 Python wheels for x86_64 and arm64 - CPU Python wheels for x86_64, arm64, and macOS - XPU Python wheel for x86_64 ## Model Support * **New models**: DeepSeek-V4.1-Flash (#56214, #56228, #56208), DeepSeek-V4-Flash-Vision-Exp (#54566) with ROCm support (#55107) and LoRA (#55897), GLM-5.3-Flash (#53906), K2-Horizon with its reasoning and tool parsers (#55063), Cohere Compass (#54774, #55949), Bailing V3 VL with MTP (#55921), Nanbeige4.2 via the Transformers backend (#56071), and a DeepSeek-V4 CPU backend (#55355). * **DeepSeek-V4.1-Flash**: whole KV in MXFP8 through the FlashMLA V4.1 record on SM100 (#56893), Mega-mHC from the DeepGEMM fork (#56962), mHC post block folded into the delayed pre projection (#56633), Triton-fused input metadata preparation (#56562), async prefetch for CPU-offloaded Engram lookups and Engram DP sharding (#56512), DSpark draft states collapsed before the SP all-gather (#56903), DSpark drafter no longer inheriting uninitialized EPLB state (#56387), XGrammar V4.1 schema constraints for strict tool parameters (#56408), Responses text parts (#56299), and image sentinel padding scoped to the Vision-Exp checkpoint (#56554). * **DeepSeek V4**: plain RoPE without YaRN on sparse SWA layers (#54815), `--kv-cache-dtype auto` resolving to `fp8_ds_mla` with FlashMLA (#45091), prefill sparse index workspace sentinel seeded (#55299), dequant gather grid sized by rows (#55061), optional Q-norm in the fused MLA epilogue and group_size=32 packed FP8 quant (#56215), sparse settings read from the text config for composite models (#56160), MegaMoE startup without EP fixed (#55914), and a Triton iHC pre/post fallback for HY V4 (#55059). * **GLM-5.3-Flash**: EPLB support (#55119), FlashKDA for KDA chunked prefill (1.7-3.8x faster than the Triton chunk path, #55737), dense and masked-MHA sparse prefill for the NoPE (256, 0, 256) layout (#55738), decode hot-path cleanups removing per-layer copies and a duplicate router GEMM (#55736), and the `glm5next` NVIDIA subtree now packaged in wheels (#55214). * **Qwen3.8-Flash-Next**: separate prefill and decode QSA indexer kernels with `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` (#54513), fused PLE kernels and merged K/V projections (#54517), sparse GQA skipping padded indices (#54873), FP8 indexer cache (#54890), fused PLE residual and QSA output gate (#55309), UVA PLE offload and Engram tensor parallelism via `--engram-config` (#54371), torch.compile removed from the NVIDIA path (#55272), compact indexer logits workspace (#54915), HC combine-norm reused for MTP input (#54687), PLE MTP metadata transfers (#55054), Hopper LL-GEMM tuning table (#54560), B200 FP8 MoE tuning (#55890), and fixes for FP8 PLE weight scales (#54722, #54882) and fused PLE conv strides (#55375). * **Kimi K3**: native CUDA AttnRes default on SM100 (#54261), ll_bf16 router weight prefetch (#53524), overlapped TP8 KDA projections (#54697), strided DSV3 GEMM (#54565), MLA decode epilogue latency (#54896), NVFP4 in_proj alignment routing to DeepGEMM (#55242), mixed-batch KDA without gather/scatter (#56159), grouped FP8 MLA cache insertion (#55356), FlashInfer KDA kernels (#55364), FlashKDA bf16 checkpoint state (#56485) and a stability bump (#54859), SiTU on the FlashInfer CuTeDSL MoE backend (#54606), internal prefix checkpoints with partial prefix caching and spec decoding (#53614), and fixes for RecoverSSM startup (#54634, #55774), GDN metadata (#54636), mixed-batch masks (#54781), checkpoint indexing (#55747, #56794), KDA illegal memory access (#55924), interleaved weight streams (#53379), NVFP4 MoE conversion OOM (#55407), and KDA projection overlap on Hopper (#55426). * **Transformers backend**: attention found with a fuser and vLLM's layer attached to it (#54941), generic merged-column linear fusion (#55301), QKV fuser for Gemma4 (#55690), all vocab embeddings replaced including Gemma 3n/4 per-layer tables (#54760, #55083), and attention sinks applied (#52156). * **Speculative decoding for models**: EAGLE3 for Sarvam MLA (#53052), Nemotron-H MTP with a separate, possibly quantized lm_head (#54574), NVFP4 DSpark gathered top-k projection for quantized Markov heads (#55713), Qwen3.5/3.6 multimodal MTP resolving `n_predict` from the text config (#55369), Qwen3 DSpark padded-vocab drafts (#55133), and GLM-OCR MTP weight loading (#49869) and position masking under CUDA graphs (#56447). * **Multimodal**: torchcodec audio decoding and a selectable `audio_backend` in `--media-io-kwargs` (#51826) with automatic decoding kept soundfile-first (#55642), torchaudio as the default resampler (#52598), `media_io_kwargs` in multimodal hashes (#54241), cache hash kwargs scoped by modality (#54918), empty video URLs with multimodal UUIDs (#54220), cached audio inputs with UUIDs (#56310), encoder CUDA graphs for MiniCPM-V 2.5/2.6/4.0 (#42785), Voxtral Realtime with `FULL_DECODE_ONLY` graphs (#51167), Triton/FlashInfer composite attention for multimodal prefixes (#56305), pruned sliding-window tiles for Gemma 4 multimodal prompts (up to 3-4x E2E, #53147), SDPA for BLIP-2 Q-Former (#55285), baddbmm Conformer scores (#55062), fused DeepEncoder relative bias (#55629), Gemma3n sparse GELU Triton kernel (#48498), and no duplicate text embedding in Qwen2.5-Omni (#55415). * **Correctness**: Gemma 4 / Gemma 3n fine-tuned checkpoints without KV projections on shared layers (#54917, #55864), Gemma4 video preprocessing without image kwargs (#56652) and unified audio profiling (#56721), MiniCPM-V 4.6 loading (#49417) and video embeds (#54633), InternVL video parser (#55779), Nano-Nemotron precomputed embeddings (#56398), OpenPangu multimodal merge (#55941) and sleep mode (#53696), Qwen3-VL and Cosmos3-Edge under CPU and PP (#53699, #43272), Qwen VL processors without a tokenizer (#54886), Molmo2 fps guard (#55893), CohereASR audio-token estimate (#53829), double BOS in `LLM.chat()` for multimodal models (#55288), decoded frame lists parsed as one video (#55326), swapped H/W in dummy video profiling (#56385
document fetched 2026-09-29 02:33