← sources

Aurora2-400M-A20M - a tiny, sparse MoE model.

source: reddit (reddit:LocalLLaMA)published 2026-11-18 20:52ingested 2026-11-19 02:39

retained text

It was pretrained from scratch using 100B tokens on a single RTX Pro 6000 in around 36 Hours.

Aurora2 uses 32 experts, of which 1 is active per token.

A few highlights:

12 transformer layers, 192 hidden dimensions,
2,048-token context length,
16,384-token vocabulary,
Muon + AdamW optimizers

Trained on a mixture of web, mathematics, Wikipedia, and conversational datasets.

More information as well as a quick start can be found in the hf repo. Any questions or feedback is welcome!

document fetched 2026-11-19 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xzqi92/aurora2400ma20m_a_tiny_sparse_moe_model