← sources

RAM Offloading with vLLM - tcclaviger appreciation post

source: reddit (reddit:LocalLLaMA)published 2026-09-29 17:14ingested 2026-09-30 02:44

retained text

Thanks to tcclaviger, vLLM now has expert RAM offloading support ([link](https://huggingface.co/tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ)). This makes frontier models much more accessible on a local setup!

I was able to run the original DeepSeek-V4-Flash-Vision-Exp on four R9700s.

    podman run --rm -it \
        --init \
        --network host \
        --ulimit memlock=-1:-1 \
        -v /models:/models:ro \
        -v ~/.vllm-cache:/cache \
        -e VLLM_ROCM_USE_AITER=0 \
        -e ROCR_VISIBLE_DEVICES=0,1,2,3 \
        -e VLLM_CACHE_ROOT=/cache/vllm \
        -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \
        -e TRITON_CACHE_DIR=/cache/triton \
        --device /dev/kfd \
        --device /dev/dri \
        --group-add keep-groups \
        --annotation run.oci.keep_original_groups=1 \
        --security-opt label=disable \
        --security-opt seccomp=unconfined \
        --shm-size 160g \
        docker.io/tcclaviger/vllm@sha256:ef99b3d07c3f15e7978528c7510762ba024df9ab4242070d8ed092cd4cc1a694 \
        /models/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
        --served-model-name DeepSeek-V4-Flash-Vision-Exp \
        --tensor-parallel-size 4 \
        --enable-expert-offload \
        --expert-offload-mem 160 \
        --reasoning-parser deepseek_v4 \
        --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
        --tool-call-parser deepseek_v4 \
        --enable-auto-tool-choice \
        --max-num-seqs 8 \
        --enable-prefix-caching \
        --enable-chunked-prefill \
        --max-num-batched-tokens 2048 \
        --kv-cache-dtype fp8 \
        --block-size 256 \
        --max-model-len 256000 \
        --gpu-memory-utilization 0.97 \
        --mm-processor-cache-gb 4.0 \
        --override-generation-config '{"max_tokens": 128000, "temperature": 1.0, "top_p": 0.95}' \
        --speculative-config '{"method":"dspark","model":"/models/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":false}' \
        --compilation-config '{"cudagraph_capture_sizes": [4,8,12,16], "max_cudagraph_capture_size": 16}' \
        --host 0.0.0.0 \
        --port 8090

document fetched 2026-09-30 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1wtg12r/ram_offloading_with_vllm_tcclaviger_appreciation