# Adaptive Streaming Optimizer: Run any Big MoE model on any device, any OS, any HW, local OpenAI-compatible server. Android, Windows, Mac, IOS, Linux. CPU, GPU, NPU (35B on phone 12gb RAM 10 tok/s decode 100-300+ prefill, Windows, Mac even more: 25+ tok/s 16 gb RAM) — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/dce97f0d0876c0f8

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-20 11:15 ingested 2026-10-21 02:41

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xapdfy/adaptive_streaming_optimizer_run_any_big_moe/)

## retained text



```
I am working on making **AI accessible by running large MoE models on affordable, consumer-grade hardware that people already own: smartphones, laptops,normal desktops.** Models several times bigger than RAM, no big GPU needed.

Disclosure: I’m the author of BigMoeOnEdge. All numbers below are real runs on my own or contributors devices, same engine.

My vision? To act as **a layer on top of llama.cpp for streaming MoE** models. I’ve developed the concept of **"adaptive streaming Planner" tailored specifically to the hardware, OS, CPU, GPU, and NPU in use.** The algorithm assesses the environment to determine the optimal allocation of resources, with the goals of:

1. **maximizing prefill and decoding speeds, and**
2. **optimizing memory management without overloading the operating system.**

Furthermore, the **OpenAI-compatible server** enables the use of local AI on consumer-grade hardware; I’ve personally used it with OpenCode - first running it on Android, and later on a Mac.

**What the planner alone does (16 GB Mac, Qwen3.6-35B-A3B Q4\_K\_M)**

||Before|With --auto (Planner|
|:-|:-|:-|
|Decode|7 tok/s|25-30 tok/s|
|2000-token prompt|21 s on CPU|3.4 s on GPU|

**By device - but you can run any MoE gguf and quant supported by llama.cpp**

|Model|On disk|Device|× its RAM|Decode|
|:-|:-|:-|:-|:-|
|Qwen3.6-35B-A3B|21 GB|12 GB Android phone|1.9×|10 tok/s 100-300+ tok/s prefill (NPU)|
|Qwen3.6-35B-A3B|21 GB|16 GB Mac|1.3×|30+ tok/s 400 tok/s prefill (GPU Metal)|
|Qwen3.6-35B-A3B|21 GB|16 GB Windows laptop|1,3x|5 tok/s|
|Qwen3-Coder-30B-A3B|18.6 GB|16 GB Mac|1.2×|23 tok/s, 10 inside opencode|
|Qwen3.8-Flash-Next 125B|76 GB|16 GB Mac|4.8×|6.5-9 tok/s|
|Qwen3.8-Flash-Next 125B|80 GB|12 GB Android phone|6.7×|2.2-3.5 tok/s|
|DeepSeek V4 Flash 284B|91 GB|12 GB Android phone|7.6×|1 tok/s|

I’d be happy to hear your thoughts on this and whether it might be useful.

I’m considering opening a PR on llama.cpp to integrate it, but I’m not sure if that aligns with the goals of their project.
```



document fetched 2026-10-21 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xapdfy/adaptive_streaming_optimizer_run_any_big_moe
