← sources

RustingStrata vs llama.cpp, DeepSeek-V4.1-Flash Q2 (249 GB) on a single RTX 3060 12 GB with 30 GB RAM

source: reddit (reddit:LocalLLaMA)published 2026-11-09 16:27ingested 2026-11-10 02:42

retained text

**RustingStrata now runs DeepSeek-V4.1-Flash (249 GB) on a single RTX 3060 12 GB with 30 GB RAM**

RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2\_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.

The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.

||RustingStrata|llama.cpp b12116 (`--cpu-moe`)|
|:-|:-|:-|
|Decode|2-3.6 tok/s|0.3-1.5 tok/s|
|Prefill|\~2 tok/s short prompts, \~0.5 tok/s at 1400 tokens|0.5-0.9 tok/s|
|Top-1 agreement with llama.cpp (teacher-forced)|189/192 on short prompts, 21/23 on a 1425-token prompt|\-|
|Sampling, tool calls|not yet|yes|

Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.

Caveats:

* Decode speed varies a lot from run to run with the page cache, because the model is much bigger than RAM.
* Greedy decoding only. The context cap is 4096.

Usage:

    strata chat     --model $M
    strata generate --model $M --prompt "The capital of France is" -n 64
    strata serve    --model $M --addr 127.0.0.1:8080

`serve` exposes `/v1/chat/completions` with JSON and SSE streaming.

Repo: [RustingStrata](https://github.com/RustingStudio/RustingStrata) (the DeepSeek section is in the README)

document fetched 2026-11-10 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xrw6uv/rustingstrata_vs_llamacpp_deepseekv41flash_q2_249