# [Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/1dcb684fe98adb26

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-08-24 13:54 ingested 2026-08-25 06:58

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1vx3img/paper_tomoe_converting_dense_large_language/)

## retained text



```
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1vx3img/paper_tomoe_converting_dense_large_language/"> <img alt="[Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning" src="https://external-preview.redd.it/q3evP6JeDpAC2MdSQHWYxnCYTqbJkElIQsLFqVSdkss.png?width=640&amp;crop=smart&amp;auto=webp&amp;s=de730fbf7ecace6df0036b21470c16a2d4feacfb" title="[Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning" /> </a> </td><td> <!-- SC_OFF --><div class="md"><p><strong>ToMoE</strong>: <strong>Converting Dense Large Language Models to Mixture-of-Experts</strong> through Dynamic Structural Pruning</p> <blockquote> <p>Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.</p> </blockquote> <ul> <li><strong>arXiv</strong> : <a href="https://arxiv.org/abs/2501.15316">https://arxiv.org/abs/2501.15316</a></li> <li><strong>Full Paper PDF</strong> : <a href="https://arxiv.org/pdf/2501.15316">https://arxiv.org/pdf/2501.15316</a></li> <li><strong>GitHub</strong> : <a href="https://github.com/gaosh/ToMoE">https://github.com/gaosh/ToMoE</a> (<strong>Yes, Code available for this</strong>)</li> <li><strong>ICML</strong> : <a href="https://icml.cc/virtual/2026/poster/68821">https://icml.cc/virtual/2026/poster/68821</a></li> <li><strong>OpenReview</strong> : <a href="https://openreview.net/forum?id=RFHq46pjb6">https://openreview.net/forum?id=RFHq46pjb6</a></li> </ul> <p>Somebody please create MOE models of recent Dense models like Qwen3.8-27B, Muse-Glimmer-30B, etc.,</p> <p>Thanks <a href="/u/KSAM-The-Randomizer">u/KSAM-The-Randomizer</a> for sharing <a href="https://www.reddit.com/r/LocalLLaMA/comments/1pfxrv5/comment/p5l0woq/?utm_source=share&amp;utm_medium=web3x&amp;utm_name=web3xcss&amp;utm_term=1&amp;utm_content=share_button">this</a> on my old thread.</p> </div><!-- SC_ON --> &#32; submitted by &#32; <a href="https://www.reddit.com/user/pmttyji"> /u/pmttyji </a> <br /> <span><a href="https://arxiv.org/abs/2501.15316">[link]</a></span> &#32; <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1vx3img/paper_tomoe_converting_dense_large_language/">[comments]</a></span> </td></tr></table>
```



document fetched 2026-08-25 06:58 · canonical https://reddit.com/r/LocalLLaMA/comments/1vx3img/paper_tomoe_converting_dense_large_language
