← sources

35B MoE running at 20.5 tok/s on an iPhone, without skipping experts

source: reddit (reddit:LocalLLaMA)published 2026-10-21 13:12ingested 2026-10-22 02:43

retained text

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called **Lookahead Expert Scheduling (LES)** to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts.

The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first.

Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves:

* **20.51 tok/s** on iPhone 18 Pro Max
* **6.07× faster** than the 3.38 tok/s baseline
* **26.7% smaller** expert weights with only a 0.16-point drop across five benchmarks
* Around **3 GB peak memory** in a low-memory configuration

The implementation, including expert scheduling and DSpark integration, is open source in [llama.cpp-omni](https://github.com/OpenBMB/llama.cpp-omni).

The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too.

document fetched 2026-10-22 00:45 · canonical https://reddit.com/r/LocalLLaMA/comments/1xbmsgf/35b_moe_running_at_205_toks_on_an_iphone_without