# Aurora2-400M-A20M - a tiny, sparse MoE model. — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/ce654e05e0876b4d

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-18 20:52 ingested 2026-11-19 02:39

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xzqi92/aurora2400ma20m_a_tiny_sparse_moe_model/)

## retained text



```
It was pretrained from scratch using 100B tokens on a single RTX Pro 6000 in around 36 Hours.

Aurora2 uses 32 experts, of which 1 is active per token.

A few highlights:

12 transformer layers, 192 hidden dimensions,
2,048-token context length,
16,384-token vocabulary,
Muon + AdamW optimizers

Trained on a mixture of web, mathematics, Wikipedia, and conversational datasets.

More information as well as a quick start can be found in the hf repo. Any questions or feedback is welcome!
```



document fetched 2026-11-19 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xzqi92/aurora2400ma20m_a_tiny_sparse_moe_model
