35B MoE running at 20.5 tok/s on an iPhone, without skipping experts
retained text
I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called **Lookahead Expert Scheduling (LES)** to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts. The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first. Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves: * **20.51 tok/s** on iPhone 18 Pro Max * **6.07× faster** than the 3.38 tok/s baseline * **26.7% smaller** expert weights with only a 0.16-point drop across five benchmarks * Around **3 GB peak memory** in a low-memory configuration The implementation, including expert scheduling and DSpark integration, is open source in [llama.cpp-omni](https://github.com/OpenBMB/llama.cpp-omni). The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too.
document fetched 2026-10-22 00:45 · canonical https://reddit.com/r/LocalLLaMA/comments/1xbmsgf/35b_moe_running_at_205_toks_on_an_iphone_without