← sources

Getting incredible results with a single RTX 3090ti and 128gb of ram. Deepseek-V4.1-Flash-1017 (35 tok/s decode) and Qwen3.8-Flash-Next (130 tok/s decode)

source: reddit (reddit:LocalLLaMA)published 2026-11-01 21:01ingested 2026-11-02 02:41

extract

I'm sharing this here because this is working way better than it used to a few months ago with older models. I see people here posting slower speeds with much better hardware than mine and hopefully this information helps. I used gemini-flash to help me setup everything and I asked it to write up the sequence of steps to get me here - happy to provide more details if there's anything missing. Since I use gentoo some tools/commands might be unfamiliar to you. # My 6 year old hardware: * **OS & CPU** Gentoo Linux running on an AMD Ryzen Threadripper 3990X (64 cores, 128 threads). * **System Memory:** 128 GB DDR4 RAM. * **GPU:** 1 NVIDIA GeForce RTX 3090 Ti (24 GB GDDR6X VRAM, PCIe Gen 4 x16). * **Primary Storage:** 4 TB SSD NVMe. # Models Used, Download Sources, and Measured Speeds |**Model**|**Architecture**|**Quantization / Files**|**Download Source**|**Real Measured Speed**| |:-|:-|:-|:-|:-| |**DeepSeek-V4.1-Flash-1017**|552B MoE (40 layers, 384 experts/layer, \~39B active)|**XQ / GSQ V2** \+ `rank-1017-085.bin` \+ FP4 KV + Engram NVMe tables|[IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored](https://huggingface.co/IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored)|**\~35 tok/s decode** (with MTP/DSpark active)| |**Qwen3.8-Flash-Next**|125B MoE + 51B N-gram (GDN + QSA hybrid, \~3.5B active)|**pack-qfn45** (4-bit intermediate, DeltaNet linear KV)|\[IndrasMirror/Qwen3.8-Flash-Next-Swift-1.5-XQ-Uncensored\](https://huggingface.co/IndrasMirror/Qwen3.8-Flash-Next-Swift-1.5-XQ-Uncensored)|**\~138–140 tok/s decode**| # **IMPORTANT: NVIDIA GPU Unlock** Consumer Ampere cards (RTX 3000 series: GA102, GA104) feature artificial firmware throttles on FP16/BF16 matrix instruction issue rates compared to their enterprise workstation counterparts. The open-source `gpu-unlock` kernel module lifts this throttle prior to driver attachment. # **1. Compile and Install the Kernel Module** As root: git clone https://github.com/Zagitta/gpu-unlock.git /usr/src/gpu-unlock cd /usr/src/gpu-unlock make -j$(nproc) make install **2. Pass the Unlock Flag to the Kernel Command Line** Add `gpu_unlock.targets=all` to your bootloader kernel parameters (e.g. in `/etc/default/grub` under `GRUB_CMDLINE_LINUX_DEFAULT`, or your direct EFISTUB / systemd-boot entry): gpu_unlock.targets=all Regenerate your boot configuration and rebuild your initramfs so the module is included and loads before the NVIDIA kernel modules: # If using GRUB: grub-mkconfig -o /boot/grub/grub.cfg # If using dracut: dracut -f --regenerate-all **3. NVIDIA Driver Options & Persistence** Configure NVIDIA kernel driver parameters in `/etc/modprobe.d/nvidia-perf.conf` (as root): cat << 'EOF' > /etc/modprobe.d/nvidia-perf.conf options nvidia NVreg_PreserveVideoMemoryAllocations=1 options nvidia NVreg_EnableStreamMemOPs=1 options nvidia-uvm uvm_perf_enable_cpu_to_gpu_dma=1 EOF After rebooting, verify that the unlock executed: dmesg | grep -i gpu_unlock Enable persistence mode and set the GPU to its maximum factory power ceiling: nvidia-smi -pm 1 nvidia-smi -pl $(nvidia-smi -q -d POWER | grep "Max Power Limit" | head -n1 | awk '{print $5}' | cut -d. -f1) **Step 2: System Dependencies & Tooling** Install build dependencies and runtime libraries: emerge sys-devel/gcc:14 # If GCC 14 is not selected, set it to this version gcc-config -l # then replace <N> with the number from gcc-config -l): # gcc-config <N> && source /etc/profile # this won't compile with GCC > 14 emerge dev-util/nvidia-cuda-toolkit # for compleness, just in case emerge --ask --verbose \ dev-build/cmake \ dev-build/ninja \ dev-vcs/git \ net-misc/curl \ app-misc/jq \ sys-libs/liburing \ app-arch/snappy \ dev-lang/python Prepare the directory structure under your user environment: mkdir -p "$HOME/.local/bin" mkdir -p "$HOME/.local/llm-tools" mkdir -p "$HOME/.quetza-data/conductor" mkdir -p "$HOME/models/deepseek-v41" mkdir -p "$HOME/models/qwen-flash" # Dedicated Python environment for tool scripts & Hugging Face downloads python3 -m venv "$HOME/.local/llm-tools/venv_hf" source "$HOME/.local/llm-tools/venv_hf/bin/activate" pip install --upgrade pip pip install "huggingface_hub[cli]" torch safetensors numpy **Step 3: Build the Xolotl Inference Engine** Xolotl integrates the multi-tier memory management, NVMe Engram streaming, and DSpark speculative decoding logic needed to run large MoE models on a single GPU. cd "$HOME/.local/llm-tools" git clone https://github.com/Indras-Mirror/Xolotl.git cd Xolotl # Fetch pinned llama.cpp dependencies and apply patches bash tools/xolotl/fetch-llama.sh # Configure and compile for Ampere (sm_86) cmake -S . -B build -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DSTRATA_ENABLE_CUDA=ON \ -DSTRATA_GGML_CUDA=ON \ -DSTRATA_MMQ_KQUANTS=ON \ -DCMAKE_CUDA_ARCHITECTURES=86 \ -DSTRATA_GGML_DIR="$PWD/third_party/llama.cpp" \ -DSTRATA_DSV41_FQ_OP=ON \ -DSTRATA_DSV41_HC_MIX=ON ninja -C build # Link the build directory so wrapper scripts resolve ds41_generate ln -sfn "$PWD/build" "$PWD/build-xolotl" **Step 4: Model Downloads & Engram Extraction** ***1. DeepSeek-V4.1-Flash (552B MoE)*** Bash source "$HOME/.local/llm-tools/venv_hf/bin/activate" cd "$HOME/models/deepseek-v41" # Download model weights, draft sidecar, and expert rank index hf download IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored \ --include "v2/*" "tokenizer/*" "V41-MTP-Vision-1017*" \ --local-dir . # Download original FP8 Engram table shards hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \ model-00047-of-00048.safetensors \ model-00048-of-00048.safetensors \ model.safetensors.index.json config.json \ --local-dir ./FP8-engram # Extract Engram runtime binary tables onto NVMe cd "$HOME/.local/llm-tools/Xolotl" python3 tools/v41/extract_engram.py \ --input "$HOME/models/deepseek-v41/FP8-engram" \ --output "$HOME/models/deepseek-v41/engram_rt" ***2. Qwen3.8-Flash-Next (125B MoE)*** cd "$HOME/models/qwen-flash" hf download Qwen/Qwen3.8-Flash-Next-GGUF \ --include "pack-qfn45/*" \ --local-dir . **Step 5: MTP Speculative Decoding Setup** MTP (Multi-Token Prediction) on DeepSeek-V4.1-Flash is driven by the **DSpark sidecar** engine: * **Draft Engine:** Uses `V41-MTP-Vision-1017.Q2_K.gguf` as the proposal head. * **Expert Pruning:** Restricts verification sweeps to the top 64 active draft experts (`dspark_expert_counts.txt:64`), preventing PCIe bus saturation. * **On-the-fly Re-quantization (**`STRATA_DENSE_REQUANT=q6_K`**):** Re-quantizes the dense attention and output heads from Q8\_0 to Q6\_K at launch. This reclaims **1.6 GiB of VRAM**, fitting the speculative draft cache alongside active experts. * **PCIe Batching (**`--tier-batch-pcie 0.25`**):** Batches DMA transfers between the 80 GiB pinned host RAM arena and the GPU. **1. Expand Process Memory Boundary in the Engine Wrapper** Allow the systemd user scope to hold the full 80 GiB arena plus runtime buffers: Bash sed -i 's/MemoryMax=84G/MemoryMax=110G/g' "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh" **2. Generate Base Configuration:** $HOME/models/deepseek-v41/strata-v41.json cat << EOF > "$HOME/models/deepseek-v41/strata-v41.json" { "exe": "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh", "args": [ "-m", "$HOME/models/deepseek-v41/v2/V41-XQ-1017.gguf", "--backend", "cuda", "--experts", "gpu", "--engram", "$HOME/models/deepseek-v41/engram_rt", "--rank", "$HOME/models/deepseek-v41/v2/rank-1017-085.bin", "--ctx", "131072", "--stop", "1", "--arena-adapt", "--kv-fp4", "--route-bias", "0.5", "--route-bias-file", "0.3", "--arena-gib", "80", "--prompt-batch", "12288", "--tier-batch-pcie", "0.25" ], "cwd": "$HOME/.local/llm-tools/Xolotl", "tokenizer": "$HOME/models/deepseek-v41/tokenizer", "model_name": "deepseek-v4.1", "aliases": "deepseek-v41,deepseek/deepseek-v4.1-flash", "log": "$HOME/models/deepseek-v41/xolotl-v41.log", "port": 8000, "gpu": 0, "sampling": { "temperature": 0.1, "top_p": 0.95, "top_k": 64, "presence_penalty": 0.0, "frequency_penalty": 0.0 }, "api_monitor": true, "fit_max_tokens": true, "reasoning_budget_tokens": 4096, "env": { "STRATA_BUILD_DIR": "$HOME/.local/llm-tools/Xolotl/build", "STRATA_SERVE_DSPARK": "$HOME/models/deepseek-v41/V41-MTP-Vision-1017.Q2_K.gguf", "STRATA_ATTN_F16_GATHER": "1", "STRATA_NO_SCOPE": "1", "STRATA_SERVE_PASS_BIAS": "0", "STRATA_DENSE_REQUANT": "q6_K", "STRATA_SERVE_LOOKUP": "14", "STRATA_PRESTAGE_GIB": "1.5" } } EOF **Step 6: Parameterized Launch Scripts** **1. DeepSeek-V4.1 Launcher:** $HOME/.local/bin/start-deepseek-custom.sh cat << 'EOF' > "$HOME/.local/bin/start-deepseek-custom.sh" #!/usr/bin/env bash set -euo pipefail MODEL_DIR="$HOME/models/deepseek-v41" XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl" BASE_CONFIG="$MODEL_DIR/strata-v41.json" RUN_CONFIG="$MODEL_DIR/strata-v41-run.json" CTX="131072" PORT="8000" TEMP="0.1" THINK_BUDGET="4096" ARENA_GIB="80" PASS_BIAS="0" PROMPT_BATCH="12288" usage() { cat << USAGE Usage: $(basename "$0") [options] Options: -c, --ctx INT Context length in tokens (default: 131072; up to 1048576) -p, --port INT Port to bind (default: 8000) -t, --temp FLOAT Sampling temperature (default: 0.1) -b, --thinking INT Reasoning budget tokens (default: 4096) -a, --arena INT Pinned RAM expert arena in GiB (default: 80) --bias 0|1 Routing bias: 0=exact, 1=fast biased (default: 0) --batch INT Prompt batch size (default: 12288; auto-clamps to 8192 for >=512k) -h, --help Show help message USAGE exit 0 } while [[ $# -gt 0 ]]; do case "$1" in -c|--ctx) CTX="$2"; shift 2 ;; -p|--port) PORT="$2"; shift 2 ;; -t|--temp) TEMP="$2"; shift 2 ;; -b|--thinking) THINK_BUDGET="$2"; shift 2 ;; -a|--arena) ARENA_GIB="$2"; shift 2 ;; --bias) PASS_BIAS="$2"; shift 2 ;; --batch) PROMPT_BATCH="$2"; shift 2 ;; -h|--help) usage ;; *) echo "Unknown option: $1" >&2; exit 1 ;; esac done if [ "$CTX" -ge 524288 ] && [ "$PROMPT_BATCH" -gt 8192 ]; then echo "[-] Large context specified ($CTX tokens). Clamping --prompt-batch to 8192 to prevent graph OOM." PROMPT_BATCH=8192 fi python3 - << PY import json with open("$BASE_CONFIG", "r") as f: cfg = json.load(f) cfg["port"] = int("$PORT") cfg["max_context"] = int("$CTX") cfg["reasoning_budget_tokens"] = int("$THINK_BUDGET") cfg["sampling"]["temperature"] = float("$TEMP") args = cfg.get("args", []) def set_arg(flag, val): for i in range(len(args) - 1): if args[i] == flag: args[i+1] = str(val) return args.extend([flag, str(val)]) set_arg("--ctx", "$CTX") set_arg("--arena-gib", "$ARENA_GIB") set_arg("--prompt-batch", "$PROMPT_BATCH") cfg["args"] = args env = cfg.setdefault("env", {}) env["STRATA_SERVE_PASS_BIAS"] = "$PASS_BIAS" env["STRATA_PRESTAGE_GIB"] = "1.0" if int("$CTX") >= 524288 else "1.5" with open("$RUN_CONFIG", "w") as f: json.dump(cfg, f, indent=2) PY source "$HOME/.local/llm-tools/venv_hf/bin/activate" cd "$XOLOTL_DIR" echo "==> Launching DeepSeek-V4.1-Flash (552B MoE)..." echo " Context : $CTX tokens" echo " Port : $PORT" echo " Arena RAM : $ARENA_GIB GiB" echo " Prompt Batch : $PROMPT_BATCH" exec python3 tools/v41/serve/serve_ds41.py \ --engine strata \ --config "$RUN_CONFIG" \ --port "$PORT" \ --host 0.0.0.0 EOF chmod +x "$HOME/.local/bin/start-deepseek-custom.sh" **2. Qwen3.8-Flash-Next Launcher: $HOME/.local/bin/start-qwen.sh** cat << 'EOF' > "$HOME/.local/bin/start-qwen.sh" #!/usr/bin/env bash set -euo pipefail XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl" MODEL_DIR="$HOME/models/qwen-flash/pack-qfn45" PORT="8099" source "$HOME/.local/llm-tools/venv_hf/bin/activate" cd "$XOLOTL_DIR" echo "==> Launching Qwen3.8-Flash-Next (125B MoE)..." exec python3 -m serve.server \ --engine strata \ --config "$MODEL_DIR/strata-qwen.json" \ --port "$PORT" \ --host 0.0.0.0 EOF chmod +x "$HOME/.local/bin/start-qwen.sh" **Step 7: Run** # Start DeepSeek-V4.1-Flash on :8000 (default 128k context, 1M context work just fine too) $HOME/.local/bin/start-deepseek-custom.sh # Or start Qwen3.8-Flash-Next on :8099 $HOME/.local/bin/start-qwen.sh

retained text (truncated)

I'm sharing this here because this is working way better than it used to a few months ago with older models. I see people here posting slower speeds with much better hardware than mine and hopefully this information helps.

I used gemini-flash to help me setup everything and I asked it to write up the sequence of steps to get me here - happy to provide more details if there's anything missing.

Since I use gentoo some tools/commands might be unfamiliar to you.

# My 6 year old hardware:

* **OS & CPU** Gentoo Linux running on an AMD Ryzen Threadripper 3990X (64 cores, 128 threads).
* **System Memory:** 128 GB DDR4 RAM.
* **GPU:** 1 NVIDIA GeForce RTX 3090 Ti (24 GB GDDR6X VRAM, PCIe Gen 4 x16).
* **Primary Storage:** 4 TB SSD NVMe.

# Models Used, Download Sources, and Measured Speeds

|**Model**|**Architecture**|**Quantization / Files**|**Download Source**|**Real Measured Speed**|
|:-|:-|:-|:-|:-|
|**DeepSeek-V4.1-Flash-1017**|552B MoE (40 layers, 384 experts/layer, \~39B active)|**XQ / GSQ V2** \+ `rank-1017-085.bin` \+ FP4 KV + Engram NVMe tables|[IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored](https://huggingface.co/IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored)|**\~35 tok/s decode** (with MTP/DSpark active)|
|**Qwen3.8-Flash-Next**|125B MoE + 51B N-gram (GDN + QSA hybrid, \~3.5B active)|**pack-qfn45** (4-bit intermediate, DeltaNet linear KV)|\[IndrasMirror/Qwen3.8-Flash-Next-Swift-1.5-XQ-Uncensored\](https://huggingface.co/IndrasMirror/Qwen3.8-Flash-Next-Swift-1.5-XQ-Uncensored)|**\~138–140 tok/s decode**|

# 

**IMPORTANT: NVIDIA GPU Unlock**

Consumer Ampere cards (RTX 3000 series: GA102, GA104) feature artificial firmware throttles on FP16/BF16 matrix instruction issue rates compared to their enterprise workstation counterparts. The open-source `gpu-unlock` kernel module lifts this throttle prior to driver attachment.

# 

**1. Compile and Install the Kernel Module**

As root:

    git clone https://github.com/Zagitta/gpu-unlock.git /usr/src/gpu-unlock
    cd /usr/src/gpu-unlock
    make -j$(nproc)
    make install

**2. Pass the Unlock Flag to the Kernel Command Line**

Add `gpu_unlock.targets=all` to your bootloader kernel parameters (e.g. in `/etc/default/grub` under `GRUB_CMDLINE_LINUX_DEFAULT`, or your direct EFISTUB / systemd-boot entry):

    gpu_unlock.targets=all

Regenerate your boot configuration and rebuild your initramfs so the module is included and loads before the NVIDIA kernel modules:

    # If using GRUB:
    grub-mkconfig -o /boot/grub/grub.cfg
    
    # If using dracut:
    dracut -f --regenerate-all

**3. NVIDIA Driver Options & Persistence**

Configure NVIDIA kernel driver parameters in `/etc/modprobe.d/nvidia-perf.conf` (as root):

    cat << 'EOF' > /etc/modprobe.d/nvidia-perf.conf
    options nvidia NVreg_PreserveVideoMemoryAllocations=1
    options nvidia NVreg_EnableStreamMemOPs=1
    options nvidia-uvm uvm_perf_enable_cpu_to_gpu_dma=1
    EOF

After rebooting, verify that the unlock executed:

    dmesg | grep -i gpu_unlock

Enable persistence mode and set the GPU to its maximum factory power ceiling:

    nvidia-smi -pm 1
    nvidia-smi -pl $(nvidia-smi -q -d POWER | grep "Max Power Limit" | head -n1 | awk '{print $5}' | cut -d. -f1)

**Step 2: System Dependencies & Tooling**

Install build dependencies and runtime libraries:

    emerge sys-devel/gcc:14 
    
    
    # If GCC 14 is not selected, set it to this version
    gcc-config -l 
    # then replace <N> with the number from  gcc-config -l): 
    # gcc-config <N> && source /etc/profile
    
    # this won't compile with GCC > 14
    emerge dev-util/nvidia-cuda-toolkit
    
    # for compleness, just in case
    
    emerge --ask --verbose \
        dev-build/cmake \
        dev-build/ninja \
        dev-vcs/git \
        net-misc/curl \
        app-misc/jq \
        sys-libs/liburing \
        app-arch/snappy \
        dev-lang/python

Prepare the directory structure under your user environment:

    mkdir -p "$HOME/.local/bin"
    mkdir -p "$HOME/.local/llm-tools"
    mkdir -p "$HOME/.quetza-data/conductor"
    mkdir -p "$HOME/models/deepseek-v41"
    mkdir -p "$HOME/models/qwen-flash"
    
    # Dedicated Python environment for tool scripts & Hugging Face downloads
    python3 -m venv "$HOME/.local/llm-tools/venv_hf"
    source "$HOME/.local/llm-tools/venv_hf/bin/activate"
    pip install --upgrade pip
    pip install "huggingface_hub[cli]" torch safetensors numpy

**Step 3: Build the Xolotl Inference Engine**

Xolotl integrates the multi-tier memory management, NVMe Engram streaming, and DSpark speculative decoding logic needed to run large MoE models on a single GPU.

    cd "$HOME/.local/llm-tools"
    git clone https://github.com/Indras-Mirror/Xolotl.git
    cd Xolotl
    
    # Fetch pinned llama.cpp dependencies and apply patches
    bash tools/xolotl/fetch-llama.sh
    
    # Configure and compile for Ampere (sm_86)
    cmake -S . -B build -G Ninja \
        -DCMAKE_BUILD_TYPE=Release \
        -DSTRATA_ENABLE_CUDA=ON \
        -DSTRATA_GGML_CUDA=ON \
        -DSTRATA_MMQ_KQUANTS=ON \
        -DCMAKE_CUDA_ARCHITECTURES=86 \
        -DSTRATA_GGML_DIR="$PWD/third_party/llama.cpp" \
        -DSTRATA_DSV41_FQ_OP=ON \
        -DSTRATA_DSV41_HC_MIX=ON
    
    ninja -C build
    
    # Link the build directory so wrapper scripts resolve ds41_generate
    ln -sfn "$PWD/build" "$PWD/build-xolotl"

**Step 4: Model Downloads & Engram Extraction**

***1. DeepSeek-V4.1-Flash (552B MoE)***

Bash

    source "$HOME/.local/llm-tools/venv_hf/bin/activate"
    cd "$HOME/models/deepseek-v41"
    
    # Download model weights, draft sidecar, and expert rank index
    hf download IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored \
        --include "v2/*" "tokenizer/*" "V41-MTP-Vision-1017*" \
        --local-dir .
    
    # Download original FP8 Engram table shards
    hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \
        model-00047-of-00048.safetensors \
        model-00048-of-00048.safetensors \
        model.safetensors.index.json config.json \
        --local-dir ./FP8-engram
    
    # Extract Engram runtime binary tables onto NVMe
    cd "$HOME/.local/llm-tools/Xolotl"
    python3 tools/v41/extract_engram.py \
        --input "$HOME/models/deepseek-v41/FP8-engram" \
        --output "$HOME/models/deepseek-v41/engram_rt"

***2. Qwen3.8-Flash-Next (125B MoE)***

    cd "$HOME/models/qwen-flash"
    hf download Qwen/Qwen3.8-Flash-Next-GGUF \
        --include "pack-qfn45/*" \
        --local-dir .

**Step 5: MTP Speculative Decoding Setup**

MTP (Multi-Token Prediction) on DeepSeek-V4.1-Flash is driven by the **DSpark sidecar** engine:

* **Draft Engine:** Uses `V41-MTP-Vision-1017.Q2_K.gguf` as the proposal head.
* **Expert Pruning:** Restricts verification sweeps to the top 64 active draft experts (`dspark_expert_counts.txt:64`), preventing PCIe bus saturation.
* **On-the-fly Re-quantization (**`STRATA_DENSE_REQUANT=q6_K`**):** Re-quantizes the dense attention and output heads from Q8\_0 to Q6\_K at launch. This reclaims **1.6 GiB of VRAM**, fitting the speculative draft cache alongside active experts.
* **PCIe Batching (**`--tier-batch-pcie 0.25`**):** Batches DMA transfers between the 80 GiB pinned host RAM arena and the GPU.

**1. Expand Process Memory Boundary in the Engine Wrapper**

Allow the systemd user scope to hold the full 80 GiB arena plus runtime buffers:

Bash

    sed -i 's/MemoryMax=84G/MemoryMax=110G/g' "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh"

**2. Generate Base Configuration:** $HOME/models/deepseek-v41/strata-v41.json

    cat << EOF > "$HOME/models/deepseek-v41/strata-v41.json"
    {
      "exe": "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh",
      "args": [
        "-m",
        "$HOME/models/deepseek-v41/v2/V41-XQ-1017.gguf",
        "--backend",
        "cuda",
        "--experts",
        "gpu",
        "--engram",
        "$HOME/models/deepseek-v41/engram_rt",
        "--rank",
        "$HOME/models/deepseek-v41/v2/rank-1017-085.bin",
        "--ctx",
        "131072",
        "--stop",
        "1",
        "--arena-adapt",
        "--kv-fp4",
        "--route-bias",
        "0.5",
        "--route-bias-file",
        "0.3",
        "--arena-gib",
        "80",
        "--prompt-batch",
        "12288",
        "--tier-batch-pcie",
        "0.25"
      ],
      "cwd": "$HOME/.local/llm-tools/Xolotl",
      "tokenizer": "$HOME/models/deepseek-v41/tokenizer",
      "model_name": "deepseek-v4.1",
      "aliases": "deepseek-v41,deepseek/deepseek-v4.1-flash",
      "log": "$HOME/models/deepseek-v41/xolotl-v41.log",
      "port": 8000,
      "gpu": 0,
      "sampling": {
        "temperature": 0.1,
        "top_p": 0.95,
        "top_k": 64,
        "presence_penalty": 0.0,
        "frequency_penalty": 0.0
      },
      "api_monitor": true,
      "fit_max_tokens": true,
      "reasoning_budget_tokens": 4096,
      "env": {
        "STRATA_BUILD_DIR": "$HOME/.local/llm-tools/Xolotl/build",
        "STRATA_SERVE_DSPARK": "$HOME/models/deepseek-v41/V41-MTP-Vision-1017.Q2_K.gguf",
        "STRATA_ATTN_F16_GATHER": "1",
        "STRATA_NO_SCOPE": "1",
        "STRATA_SERVE_PASS_BIAS": "0",
        "STRATA_DENSE_REQUANT": "q6_K",
        "STRATA_SERVE_LOOKUP": "14",
        "STRATA_PRESTAGE_GIB": "1.5"
      }
    }
    EOF

**Step 6: Parameterized Launch Scripts**

**1. DeepSeek-V4.1 Launcher:** $HOME/.local/bin/start-deepseek-custom.sh

    cat << 'EOF' > "$HOME/.local/bin/start-deepseek-custom.sh"
    #!/usr/bin/env bash
    set -euo pipefail
    
    MODEL_DIR="$HOME/models/deepseek-v41"
    XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl"
    BASE_CONFIG="$MODEL_DIR/strata-v41.json"
    RUN_CONFIG="$MODEL_DIR/strata-v41-run.json"
    
    CTX="131072"
    PORT="8000"
    TEMP="0.1"
    THINK_BUDGET="4096"
    ARENA_GIB="80"
    PASS_BIAS="0"
    PROMPT_BATCH="12288"
    
    usage() {
      cat << USAGE
    Usage: $(basename "$0") [options]
    
    Options:
      -c, --ctx INT           Context length in tokens (default: 131072; up to 1048576)
      -p, --port INT          Port to bind (default: 8000)
      -t, --temp FLOAT        Sampling temperature (default: 0.1)
      -b, --thinking INT      Reasoning budget tokens (default: 4096)
      -a, --arena INT         Pinned RAM expert arena in GiB (default: 80)
      --bias 0|1              Routing bias: 0=exact, 1=fast biased (default: 0)
      --batch INT             Prompt batch size (default: 12288; auto-clamps to 8192 for >=512k)
      -h, --help              Show help message
    USAGE
      exit 0
    }
    
    while [[ $# -gt 0 ]]; do
      case "$1" in
        -c|--ctx)       CTX="$2"; shift 2 ;;
        -p|--port)      PORT="$2"; shift 2 ;;
        -t|--temp)      TEMP="$2"; shift 2 ;;
        -b|--thinking)  THINK_BUDGET="$2"; shift 2 ;;
        -a|--arena)     ARENA_GIB="$2"; shift 2 ;;
        --bias)         PASS_BIAS="$2"; shift 2 ;;
        --batch)        PROMPT_BATCH="$2"; shift 2 ;;
        -h|--help)      usage ;;
        *) echo "Unknown option: $1" >&2; exit 1 ;;
      esac
    done
    
    if [ "$CTX" -ge 524288 ] && [ "$PROMPT_BATCH" -gt 8192 ]; then
      echo "[-] Large context specified ($CTX tokens). Clamping --prompt-batch to 8192 to prevent graph OOM."
      PROMPT_BATCH=8192
    fi
    
    python3 - << PY
    import json
    
    with open("$BASE_CONFIG", "r") as f:
        cfg = json.load(f)
    
    cfg["port"] = int("$PORT")
    cfg["max_context"] = int("$CTX")
    cfg["reasoning_budget_tokens"] = int("$THINK_BUDGET")
    cfg["sampling"]["temperature"] = float("$TEMP")
    
    args = cfg.get("args", [])
    def set_arg(flag, val):
        for i in range(len(args) - 1):
            if args[i] == flag:
                args[i+1] = str(val)
                return
        args.extend([flag, str(val)])
    
    set_arg("--ctx", "$CTX")
    set_arg("--arena-gib", "$ARENA_GIB")
    set_arg("--prompt-batch", "$PROMPT_BATCH")
    cf

document fetched 2026-11-02 00:36 · canonical https://reddit.com/r/LocalLLaMA/comments/1xlec67/getting_incredible_results_with_a_single_rtx