Aurora2-400M-A20M - a tiny, sparse MoE model.
retained text
It was pretrained from scratch using 100B tokens on a single RTX Pro 6000 in around 36 Hours. Aurora2 uses 32 experts, of which 1 is active per token. A few highlights: 12 transformer layers, 192 hidden dimensions, 2,048-token context length, 16,384-token vocabulary, Muon + AdamW optimizers Trained on a mixture of web, mathematics, Wikipedia, and conversational datasets. More information as well as a quick start can be found in the hf repo. Any questions or feedback is welcome!
document fetched 2026-11-19 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xzqi92/aurora2400ma20m_a_tiny_sparse_moe_model