Two open-source milestones signal a shift in how large MoE models are served.
Two open-source milestones signal a shift in how large MoE models are served.
confidence score
Strong evidence: 2 independent source classes support this read.
signal brief
Two open-source milestones signal a shift in how large MoE models are served. WASTE (shared on r/LocalLLaMA) claims to run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated experts directly from NVMe, keeping only the model trunk in memory and using a bounded expert cache. If this pattern scales, it reduces the need to hold every expert in HBM/DRAM, making NVMe bandwidth a more central capacity variable in AI inference. In parallel, llama.cpp has been merging DeepSeek-specific kernels at a rapid clip: b10232 adds Metal SIMDgroup-optimized DeepSeek V4 hyper-connection ops, b10237 adds MTP support for DeepSeek V3.2, and b10231 adds DSpark sidecar resolution for speculative decoding. The burst implies nearby model releases (DeepSeek V3.2/V4, Kimi K3) are being optimized for efficiency on commodity and consumer hardware. This is an early, uncorroborated trend rather than a hard capex signal, and the net direction is mixed: bullish for NVMe/storage and local-AI tooling, potentially bearish for the assumption that HBM capacity is the only way to serve giant MoE models.
What the sources said
- WASTE README via Reddit: "It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache."
- llama.cpp b10232: "Implement GGML_OP_DSV4_HC_COMB, GGML_OP_DSV4_HC_PRE, and GGML_OP_DSV4_HC_POST with SIMDgroup register and shuffle optimized kernels."
- llama.cpp b10237: "llama : MTP support for DeepSeek V3.2 (#26457)"
- llama.cpp b10231: "dspark outranks dflash in the auto-selection since its sidecar carries the extra Markov head."
source data used
“<details open> llama : MTP support for DeepSeek V3.2 (#26457) * llama : MTP support for DeepSeek V3.2 * model : no need to include MTP layers during DeepSeek V3.2 model type discovery --------- Co-authored-by: Stanisław...”
“<details open> metal : add SILU_BACK (#25982) * feat(silu_back): implemented silu_back op for f32 * fix(silu_back): removed redundant asserts in ggml-metal-ops.cpp function ggml_metal_op_silu_back. </details> **Website:** - <https://llama.app> **macOS/iOS:** - [macOS Apple Silicon (arm64)](htt”
“<details open> metal : add F16 support for bin ops (#26465) </details> **Website:** - <https://llama.app> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b10234/llama-b10234-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enab”
“<details open> opencl: limit local workgroup size for GLU operation (#26383) </details> **Website:** - <https://llama.app> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b10233/llama-b10233-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm”
“<details open> metal: implement DeepSeek V4 hyper-connections (#26459) - Implement GGML_OP_DSV4_HC_COMB, GGML_OP_DSV4_HC_PRE, and GGML_OP_DSV4_HC_POST with SIMDgroup register and shuffle optimized kernels. - Add Metal dispatch and support plumbing and test the production Sinkhorn iteration cou”
“<details open> common: support the DSpark sidecar resolution (#26458) The dspark- files resolve like the other speculative sidecars: the -hfd tag applies to them, a requested sidecar resolves without a full model at the tag, and...”
“<details open> opencl: bugfix increment ref_count in ggml_backend_opencl_init() (#26162) Incrementing `ref_count` at the beginning is important later in the `free()` method of the `ggml_backend_opencl_context` at program end. If we do not increment the `ref_count`, the result...”
“<details open> DeepseekV4 MTP + DSpark (#25784) </details> **Website:** - <https://llama.app> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b10228/llama-b10228-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled) [DISABL”
“<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1vdy1nd/github_sqliteaiwaste_run_the_full/"> <img alt="GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embedda”
Decision support, not stock advice. This signal is research with cited evidence — not a recommendation to buy, sell, or hold any security.