# Small looped MoE design — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/d60848d4c47c28c8

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-17 16:33 ingested 2026-11-18 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xyp2ha/small_looped_moe_design/)

## retained text



```
Imagine a small MoE model like 10b a2b (numbers are from the head, ignore the parameters size) that will combine the approach of looped models + engram table for knowledge as additional 20-50b of static params that can stay on ssd and the Dories approach on the kv cache optimisatio s per loop. All of these suppose to handle the weaknesses of the MoE and looped models.

The MoE suppose to boost the models sparsity per loop so it wiil give a beeter scaling over the dense model.

The engramm table allows to add a cheap knowledge in terms of the computation so the active params stay low

The looping suppose to boost the reasoning

The nvda dorry approach fix the scaling kv cache size and all together will give the max out of the looped moe covering the weaknesses. The small active params suppose to hide the looping cost while allowing to run the models on low end/small vram cards. 

I hope that the new Qwen4 moe if they would exist will have at least some of this
```



document fetched 2026-11-18 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xyp2ha/small_looped_moe_design
