Betelgeuse: Running Qwen 3.8 Flash Next at 22 tok/s sustained (spiking to 48 tok/s) on an RTX 3060 12GB + 16GB DDR4 RAM 100% bit-exact!
retained text
About two months ago I posted here about trying to predict which MoE experts get used on the next token to speed up CPU/GPU offload on Qwen 3.6 35B. Quick confession on that older post before anything else: I ended up throwing that prototype away. To get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low). It made the tok/s counter go brrr, but on an already low-bit quant it quietly degraded output quality and reasoning. I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it. When **Qwen 3.8 Flash Next** dropped, I wanted to see if we could actually run the 68GB `GSQ-RCO-IQ2_XS` build at real usable speeds on a normal budget gaming PC—**without dropping a single expert**. For context, here is the exact box I tested this on (I actually pulled my extra RAM sticks out specifically to test what a regular 16GB PC gets): * **GPU:** RTX 3060 12GB * **RAM:** 16GB DDR4, single-channel (\~19 GB/s) * **OS:** CachyOS / Arch Linux (tested inside a strict `MemoryMax=12G` cgroup) * **Storage:** Consumer PCIe NVMe SSD (\~2.1 GB/s) If you try to load a 68GB MoE on 12GB VRAM + 16GB RAM in stock `llama.cpp`, you already know what happens: `mmap` thrashes the OS page cache to death, you get \~1,500 major page faults per token, and generation crawls at **1.47 tok/s**. Then I tested **Strata**. Strata is super cool and hits 40+ tok/s *if* you have 48GB–64GB of RAM to `mlock` 24GB+ of experts in system memory. But on a 16GB RAM setup, it refuses to load unless you force `--low-ram mmap`, where it drops down to **8.11 tok/s** because it's bound by cold SSD reads. So over the last week I built **Betelgeuse** (`bel-cli` / `bel-server`, a fork of `llama.cpp`, and I'm also opening a clean upstream PR for `llama.cpp` even if it probably won't get merged lol). # What it actually does under the hood: * **Zero gate pruning / 100% bit-exact:** Every single one of the 10 routed experts per layer across all 48 layers is computed at full precision. We verified **28,080 router records and 576 generated tokens byte-for-byte** against golden stock output (`0.000e+00` diff). * **Arrival-Slack VRAM Slot Pool (\~5,000 slots in VRAM):** All 49 dense layers + GDN + QSA + LM head stay in \~3.2 GB of VRAM. The remaining VRAM holds \~5,000 expert slots distributed non-uniformly (240 slots/layer on early layers 0–4 where there is 0ms of compute slack, 92 on middle layers, 60 on tail layers). * **In-Stream GPU Doorbell + 1-Split CUDA Graph Replay:** Instead of `ggml_backend_sched` splitting the graph 48 times per token to ask the CPU which experts missed (which burns \~360ms/token in `cudaStreamSynchronize`), a custom in-stream CUDA kernel resolves hits in \~2 µs on-device and only rings a mapped host doorbell when an expert is missing. Per-layer token embeddings (`per_layer_token_embd`) are also staged to CUDA0 so the entire decode pass runs as **1 single CUDA Graph replay**. * **RAM Victim Cache +** `io_uring` **Pager:** When an expert gets evicted from the VRAM pool, it moves into a dedicated host RAM victim cache instead of getting thrown away.VRAM and RAM stay 100% exclusive instead of duplicating the same hot experts in the OS page cache. Physical NVMe reads drop from \~71 MB/token down to **12.43 MB/token**. * **No duplicate 35GB expert bloat on disk:** Instead of keeping 35GB of dead expert tensors inside `Shard 1 .gguf` AND another 35GB in `experts.bin`, we stripped the expert payloads out of Shard 1 so the main GGUF is only **3.2 GB (**`Dense.gguf`**)** \+ **35.3 GB (**`experts.bin`**, 4KB-aligned single-read blobs)** \+ \*\*26.8 GB (\*\*`PLE.gguf` **n-gram shard)**. # Measured Results (RTX 3060 12GB + 16GB DDR4, MemoryMax=12G) |Engine|Cold Decode (Run 1)|Warm Decode (Run 2)|Peak Bursts|Prompt Processing (PP)|Physical SSD Reads|Output Parity| |:-|:-|:-|:-|:-|:-|:-| |**Stock** `llama.cpp` **(mmap)**|1.47 tok/s|1.63 tok/s|\~2 tok/s|14–20 tok/s|208–312 MiB/tok|Baseline| |**Strata (**`--low-ram mmap`**)**|8.11 tok/s|8.11 tok/s|\~9 tok/s|34.6 tok/s|\~105 ms pool wait|\~0.022 KL| |**Betelgeuse (**`bel-server`**)**|**16.80 tok/s**|**22.03 tok/s**|**Up to 48 tok/s**|**16.2 – 19.3 tok/s**|**12.43 MiB/tok**|**100% Byte-Exact**| On warm runs it holds **21.8–22.03 tok/s** sustained across long coding prompts, and when generation hits stretches where all 10 routed experts are resident in the VRAM/victim pools, it **spikes up to 48 tok/s**. # Links & What's Next * **GitHub (Betelgeuse Engine —** `bel-cli` **&** `bel-server`): `[https://github.com/Betelgeuse-AI/Betelgeuse]` * **HuggingFace (Pre-packed** `Alpha-Orionis-XS` **3.2GB Dense + Experts bundle):** `https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS` * **Upstream** `llama.cpp` **PR:** `[https://github.com/ggml-org/llama.cpp/pull/31469]` 1. Even on the massive **Q8\_K\_XL** quant (\~190GB total), bel-server runs at **8 tok/s cold and 12 tok/s warm** (with bursts up to 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 box, where stock llama.cpp immediately OOMs or crawls at <0.1 tok/s. 2. Zero experts are dropped or thresholded (0.000e+00 diff across 28,080 router records and 576 tokens vs unmodified stock). The speedup comes from eliminating the 48 per-layer CPU/GPU scheduler splits (KICK/WAIT doorbell around the shared expert), keeping the RAM victim cache strictly exclusive of VRAM (MOE\_DROP\_MMAP), and pre-staging cold misses via the upgraded oracle + 4KB-aligned io\_uring pack. **What I'm working on next:** We're currently bringing this exact streaming + victim-cache architecture over to **DeepSeek V4.1 Flash (1017)**—early testing under development is already sitting around **\~25 tok/s** on the same hardware class. Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)
document fetched 2026-11-01 00:38 · canonical https://reddit.com/r/LocalLLaMA/comments/1xk8s6z/betelgeuse_running_qwen_38_flash_next_at_22_toks