MobileMoE - a facebook Collection
retained text
<table> <tr><td> <a href="https://www.reddit.com/r/LocalLLaMA/comments/1vxbpcc/mobilemoe_a_facebook_collection/"> <img alt="MobileMoE - a facebook Collection" src="https://external-preview.redd.it/qWAEbcWZrM1iGsZ9WfpE7cv1wsUC5nIBemZJq9ltTVQ.png?width=640&crop=smart&auto=webp&s=f20ba125d55f462841ddb75430ce1203dc9e733e" title="MobileMoE - a facebook Collection" /> </a> </td><td> <!-- SC_OFF --><div class="md"><p><strong>MobileMoE</strong> is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion <em>active</em> parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a <strong>Base</strong> model (pre-training + mid-training), an <strong>SFT</strong> model (supervised fine-tuning), and a <strong>QAT</strong> model (quantization-aware training). You are currently in the <strong>MobileMoE-L-Base</strong> repository — the pre-trained 0.9B-active base model.</p> <p><strong>Model:</strong> MobileMoE-L-Base (pre-trained + mid-trained)<br /> <strong>Active Parameters:</strong> 922M<br /> <strong>Total Parameters:</strong> 5.3B<br /> <strong>Layers:</strong> 32<br /> <strong>Model Dimension:</strong> 1280<br /> <strong>Attention Heads:</strong> 20<br /> <strong>KV Heads:</strong> 4 (GQA)<br /> <strong>Head Dimension:</strong> 64<br /> <strong>Routed Experts:</strong> 60 (fine-grained, FFN hidden dim 640 each)<br /> <strong>Active Experts per Token:</strong> 4 (top-k sigmoid routing, with normalization)<br /> <strong>Shared Expert:</strong> 1, always on (FFN hidden dim 2560)<br /> <strong>Vocabulary Size:</strong> 128,256<br /> <strong>Other Features:</strong> QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)<br /> <strong>Input Modality:</strong> Text<br /> <strong>Output Modality:</strong> Text<br /> <strong>Languages:</strong> English<br /> <strong>Training Stages:</strong> Pre-training → mid-training<br /> <strong>Context Length:</strong> 8,192 tokens<br /> <strong>Precision:</strong> BF16<br /> <strong>Model Developer:</strong> Meta<br /> <strong>Model Release Date:</strong> Aug 2026<br /> <strong>License:</strong> MobileMoE is FAIR NC licensed</p> </div><!-- SC_ON -->   submitted by   <a href="https://www.reddit.com/user/jacek2023"> /u/jacek2023 </a> <br /> <span><a href="https://huggingface.co/collections/facebook/mobilemoe">[link]</a></span>   <span><a href="https://www.reddit.com/r/LocalLLaMA/comments/1vxbpcc/mobilemoe_a_facebook_collection/">[comments]</a></span> </td></tr></table>
document fetched 2026-08-25 06:58 · canonical https://reddit.com/r/LocalLLaMA/comments/1vxbpcc/mobilemoe_a_facebook_collection