# I trained LLaMa from scratch — retained record

> Retained reddit record from reddit:deeplearning.

Canonical: https://highsignal.app/data/records/6153de2be2f25e65

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:deeplearning)  published 2026-10-29 07:52 ingested 2026-10-30 02:37

[original source ↗](https://www.reddit.com/r/deeplearning/comments/1xidpzf/i_trained_llama_from_scratch/)

## retained text



```
I trained the [LLaMA model from scratch](https://github.com/CloudedLeopard17/llama-from-scratch). Implemented RoPE, grouped-query attention, RMSNorm, SwiGLU, and KV cache with batched inference.Used [Umar Jamil's](https://www.youtube.com/@umarjamilai) great video on LLaMA for learning. Used three datasets for training: 1. tiny-shakespeare (character-level), to check that everything was working fine; 2. TinyStories-v2, a 41M-parameter model; 3. FineWeb, a 125M-parameter model trained on 5.27B tokens on 2x A5000 GPUs for 11 hours. 

From few time I am also learning and implementing RL algorithms. I think the next project will be an RLHF one. But I have to look at what model I can fit on my GPU.  
```



document fetched 2026-10-30 00:24 · canonical https://reddit.com/r/deeplearning/comments/1xidpzf/i_trained_llama_from_scratch
