Small looped MoE design
retained text
Imagine a small MoE model like 10b a2b (numbers are from the head, ignore the parameters size) that will combine the approach of looped models + engram table for knowledge as additional 20-50b of static params that can stay on ssd and the Dories approach on the kv cache optimisatio s per loop. All of these suppose to handle the weaknesses of the MoE and looped models. The MoE suppose to boost the models sparsity per loop so it wiil give a beeter scaling over the dense model. The engramm table allows to add a cheap knowledge in terms of the computation so the active params stay low The looping suppose to boost the reasoning The nvda dorry approach fix the scaling kv cache size and all together will give the max out of the looped moe covering the weaknesses. The small active params suppose to hide the looping cost while allowing to run the models on low end/small vram cards. I hope that the new Qwen4 moe if they would exist will have at least some of this
document fetched 2026-11-18 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xyp2ha/small_looped_moe_design