← sources

Qwen3.8 Flash Next 125B IQ3_XXS on a 24GB M4 Pro: 7.5GB peak process memory at 10 t/s

source: reddit (reddit:LocalLLaMA)published 2026-10-30 15:30ingested 2026-10-31 02:40

retained text

I wanted to run a larger local model on my Mac mini without having to close all other apps. With Qwen 3.8 Flash Next, I’m now getting around 10 tok/s while keeping Safari and other apps open.

The runtime I’ve been working on is called MoEShift. I’ve released the alpha as open source: [GitHub, installation and benchmarks](https://github.com/davide-troiani/moeshift).

These results are from my M4 Pro with 24GB unified memory, using ISTA-DASLab’s GSQ-RCO IQ3\_XXS quant (76GB on disk).

I compared it with Strata 0.2.7 using the same model in cold runs.   
With a 6k token prompt and 256 output tokens, **total inference took 63.3s versus Strata’s 129.9s**. Prompt processing reached 160.8 tok/s versus 61.8, with generation at 9.9 tok/s and peak memory at 7.5GB versus 14GB. With a short prompt, generation was 10.44 tok/s versus Strata’s 8.4, with peak memory at 6.4GB versus 14GB. More details in the [README](https://github.com/davide-troiani/moeshift).

MoEShift loads experts from SSD as needed, caches frequently used experts in RAM and overlaps reads with GPU computation. It builds on llama.cpp  and Lookahead Expert Scheduling.  
My changes focus on Qwen 3.8 Flash Next support, execution and memory reuse between prompt processing and generation. The original quantized weights and expert routing are preserved and there is no speculative decoding. The [architecture doc](https://github.com/davide-troiani/moeshift/blob/main/docs/architecture.md) goes into the details.

I’m curious how far MoEShift can go on other Macs. If you give it a try, post your results here.

document fetched 2026-10-31 00:45 · canonical https://reddit.com/r/LocalLLaMA/comments/1xjihlu/qwen38_flash_next_125b_iq3_xxs_on_a_24gb_m4_pro