# Curated list of resources: Theory and Code for learning Distributed Training of LLMs for beginners — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/1f233e6f169d192d

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-17 08:33 ingested 2026-11-18 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xyf7ys/curated_list_of_resources_theory_and_code_for/)

## retained text



```
[Must Read: Beginners]


What I did read, in order, for learning the basics of Distributed Training for LLMs From Scratch:


1. Building a distributed training framework from first principles by Umar Jamil on YT

2. Horovod: Fast and Easy Distributed Deep Learning in TensorFlow 

3. PyTorch Distributed: Experiences on Accelerating Data Parallel Training 

4. Large Scale Distributed Deep Networks 

5. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

6. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism 

7. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM 

8. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism 

9. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

10. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel


Link to the above resources [here](https://papercase.dev/c/distributed-training)

Also, if you need some rebalancing of the above (reordering or adding/removing resources which you can do it [here](https://papercase.dev/c/distributed-training?rebalance))
```



document fetched 2026-11-18 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xyf7ys/curated_list_of_resources_theory_and_code_for
