# 35B MoE running at 20.5 tok/s on an iPhone, without skipping experts — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/3d62ce11ef010fcd

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-21 13:12 ingested 2026-10-22 02:43

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xbmsgf/35b_moe_running_at_205_toks_on_an_iphone_without/)

## retained text



```
I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called **Lookahead Expert Scheduling (LES)** to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts.

The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first.

Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves:

* **20.51 tok/s** on iPhone 18 Pro Max
* **6.07× faster** than the 3.38 tok/s baseline
* **26.7% smaller** expert weights with only a 0.16-point drop across five benchmarks
* Around **3 GB peak memory** in a low-memory configuration

The implementation, including expert scheduling and DSpark integration, is open source in [llama.cpp-omni](https://github.com/OpenBMB/llama.cpp-omni).

The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too.


```



document fetched 2026-10-22 00:45 · canonical https://reddit.com/r/LocalLLaMA/comments/1xbmsgf/35b_moe_running_at_205_toks_on_an_iphone_without
