I trained a 3.87B MoE (1.45B active) from scratch on only 86.5B tokens
retained text
**First of all, thank you for reading.** I trained a small MoE model completely from scratch (no external base weights) and wanted to share the results + a couple of lessons. **Apex-2** \- Architecture: Decoder-only MoE, every layer is MoE (no dense layers) \- Size: 3.87B total parameters, 1.45B active per token \- 32 layers, d\_model 2048, GQA 16Q/4KV, 16 experts, top-4 \- Context: 4096 \- Tokenizer: Qwen3 (151k) \- Hugging Face: [https://huggingface.co/YOON1v/Apex-2](https://huggingface.co/YOON1v/Apex-2) (loads with transformers / vLLM via Qwen3MoeForCausalLM mapping) **Training** \- Pretrain: 86.5B tokens (GH200 ×1 → ×2 with **DiLoCo**) \- SFT: \~2.5B tokens (code-heavy + math + instruction) \- DPO: tried it, scores dropped, so I dropped the checkpoint **Key numbers (SFT, greedy, chat template)** **Benchmark** HumanEval 43.9 HumanEval+ 41.5 MBPP 56.3 MBPP+ 48.9 GSM8K (0-shot CoT) 32.4 MATH-500 21.0 IFEval (prompt strict) 44.7 MMLU (5-shot) 28.6 **interesting comparison** With only \~0.087T pretrain tokens, the base model’s HumanEval+ matched Qwen2.5-1.5B (which used 18T). Knowledge (MMLU) and math still lag far behind, as expected with the data gap. **What didn’t work** DPO (220k pairs, length-normalized) made answers much longer and hurt code / math / IFEval. I stopped it and kept the SFT checkpoint. Full log is in the benchmark write-up. **Limitations (honest)** \- English-centric (almost no multilingual ability) \- Weak knowledge → frequent hallucinations \- LiveCodeBench medium/hard is near zero \- 4k context only Happy to answer questions about the MoE setup, DiLoCo, or why DPO backfired.
document fetched 2026-10-05 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1wxiy8y/i_trained_a_387b_moe_145b_active_from_scratch_on