I trained LLaMa from scratch
retained text
I trained the [LLaMA model from scratch](https://github.com/CloudedLeopard17/llama-from-scratch). Implemented RoPE, grouped-query attention, RMSNorm, SwiGLU, and KV cache with batched inference.Used [Umar Jamil's](https://www.youtube.com/@umarjamilai) great video on LLaMA for learning. Used three datasets for training: 1. tiny-shakespeare (character-level), to check that everything was working fine; 2. TinyStories-v2, a 41M-parameter model; 3. FineWeb, a 125M-parameter model trained on 5.27B tokens on 2x A5000 GPUs for 11 hours. From few time I am also learning and implementing RL algorithms. I think the next project will be an RLHF one. But I have to look at what model I can fit on my GPU.
document fetched 2026-10-30 00:24 · canonical https://reddit.com/r/deeplearning/comments/1xidpzf/i_trained_llama_from_scratch