← sources

Small looped MoE design

source: reddit (reddit:LocalLLaMA)published 2026-11-17 16:33ingested 2026-11-18 02:42

retained text

Imagine a small MoE model like 10b a2b (numbers are from the head, ignore the parameters size) that will combine the approach of looped models + engram table for knowledge as additional 20-50b of static params that can stay on ssd and the Dories approach on the kv cache optimisatio s per loop. All of these suppose to handle the weaknesses of the MoE and looped models.

The MoE suppose to boost the models sparsity per loop so it wiil give a beeter scaling over the dense model.

The engramm table allows to add a cheap knowledge in terms of the computation so the active params stay low

The looping suppose to boost the reasoning

The nvda dorry approach fix the scaling kv cache size and all together will give the max out of the looped moe covering the weaknesses. The small active params suppose to hide the looping cost while allowing to run the models on low end/small vram cards. 

I hope that the new Qwen4 moe if they would exist will have at least some of this

document fetched 2026-11-18 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xyp2ha/small_looped_moe_design