Curated list of resources: Theory and Code for learning Distributed Training of LLMs for beginners
retained text
[Must Read: Beginners] What I did read, in order, for learning the basics of Distributed Training for LLMs From Scratch: 1. Building a distributed training framework from first principles by Umar Jamil on YT 2. Horovod: Fast and Easy Distributed Deep Learning in TensorFlow 3. PyTorch Distributed: Experiences on Accelerating Data Parallel Training 4. Large Scale Distributed Deep Networks 5. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models 6. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism 7. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM 8. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism 9. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning 10. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel Link to the above resources [here](https://papercase.dev/c/distributed-training) Also, if you need some rebalancing of the above (reordering or adding/removing resources which you can do it [here](https://papercase.dev/c/distributed-training?rebalance))
document fetched 2026-11-18 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xyf7ys/curated_list_of_resources_theory_and_code_for