← sources

I trained LLaMa from scratch

source: reddit (reddit:deeplearning)published 2026-10-29 07:52ingested 2026-10-30 02:37

retained text

I trained the [LLaMA model from scratch](https://github.com/CloudedLeopard17/llama-from-scratch). Implemented RoPE, grouped-query attention, RMSNorm, SwiGLU, and KV cache with batched inference.Used [Umar Jamil's](https://www.youtube.com/@umarjamilai) great video on LLaMA for learning. Used three datasets for training: 1. tiny-shakespeare (character-level), to check that everything was working fine; 2. TinyStories-v2, a 41M-parameter model; 3. FineWeb, a 125M-parameter model trained on 5.27B tokens on 2x A5000 GPUs for 11 hours. 

From few time I am also learning and implementing RL algorithms. I think the next project will be an RLHF one. But I have to look at what model I can fit on my GPU.  

document fetched 2026-10-30 00:24 · canonical https://reddit.com/r/deeplearning/comments/1xidpzf/i_trained_llama_from_scratch