Adaptive Streaming Optimizer: Run any Big MoE model on any device, any OS, any HW, local OpenAI-compatible server. Android, Windows, Mac, IOS, Linux. CPU, GPU, NPU (35B on phone 12gb RAM 10 tok/s decode 100-300+ prefill, Windows, Mac even more: 25+ tok/s 16 gb RAM)
retained text
I am working on making **AI accessible by running large MoE models on affordable, consumer-grade hardware that people already own: smartphones, laptops,normal desktops.** Models several times bigger than RAM, no big GPU needed. Disclosure: I’m the author of BigMoeOnEdge. All numbers below are real runs on my own or contributors devices, same engine. My vision? To act as **a layer on top of llama.cpp for streaming MoE** models. I’ve developed the concept of **"adaptive streaming Planner" tailored specifically to the hardware, OS, CPU, GPU, and NPU in use.** The algorithm assesses the environment to determine the optimal allocation of resources, with the goals of: 1. **maximizing prefill and decoding speeds, and** 2. **optimizing memory management without overloading the operating system.** Furthermore, the **OpenAI-compatible server** enables the use of local AI on consumer-grade hardware; I’ve personally used it with OpenCode - first running it on Android, and later on a Mac. **What the planner alone does (16 GB Mac, Qwen3.6-35B-A3B Q4\_K\_M)** ||Before|With --auto (Planner| |:-|:-|:-| |Decode|7 tok/s|25-30 tok/s| |2000-token prompt|21 s on CPU|3.4 s on GPU| **By device - but you can run any MoE gguf and quant supported by llama.cpp** |Model|On disk|Device|× its RAM|Decode| |:-|:-|:-|:-|:-| |Qwen3.6-35B-A3B|21 GB|12 GB Android phone|1.9×|10 tok/s 100-300+ tok/s prefill (NPU)| |Qwen3.6-35B-A3B|21 GB|16 GB Mac|1.3×|30+ tok/s 400 tok/s prefill (GPU Metal)| |Qwen3.6-35B-A3B|21 GB|16 GB Windows laptop|1,3x|5 tok/s| |Qwen3-Coder-30B-A3B|18.6 GB|16 GB Mac|1.2×|23 tok/s, 10 inside opencode| |Qwen3.8-Flash-Next 125B|76 GB|16 GB Mac|4.8×|6.5-9 tok/s| |Qwen3.8-Flash-Next 125B|80 GB|12 GB Android phone|6.7×|2.2-3.5 tok/s| |DeepSeek V4 Flash 284B|91 GB|12 GB Android phone|7.6×|1 tok/s| I’d be happy to hear your thoughts on this and whether it might be useful. I’m considering opening a PR on llama.cpp to integrate it, but I’m not sure if that aligns with the goals of their project.
document fetched 2026-10-21 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xapdfy/adaptive_streaming_optimizer_run_any_big_moe