Distributed Training

Jump to bottom

Sherlock edited this page Mar 12, 2021 · 1 revision

Prerequisites: NCCL

Read https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html

Good to know: MPI

Good read: https://mpitutorial.com/tutorials/

Data Parallelism

Understand NCCLAllReduce
Get familiar with DDP usage/setup

Megatron

Read https://arxiv.org/abs/1909.08053

Zero

Read https://arxiv.org/abs/1910.02054
Zero-1
- Understand ReduceScatter/AllGather
- Understand how optimizer state is partitioned
Zero-2
Zero-3

Mix of Experts

Understand All2All

Pipeline Parallelism