11. Distributed Training

Chapter focus: The "systems engineering" chapter of LLM training. Parallelism strategy sets the ceiling on compute utilization; the network and storage decide whether it can be realized. This chapter is organized into three layers: parallelism (how to partition), communication (how to connect), and stability & throughput (how to run fast and stay stable).

Articles

1. Distributed Parallelism Overview: From Single GPU to 10K — Data / Tensor / Pipeline / Sequence parallelism, ZeRO and MoE expert parallelism selection logic, with typical 7B / 70B / 1000-GPU LLM training configurations.
2. NCCL Tuning in Practice: Locating Communication Bottlenecks in a 10K-GPU Cluster — A complete troubleshooting path from nccl-tests baselines and NCCL_* environment variables to InfiniBand-side PFC/ECN configuration.
3. Mixed Precision Training & Checkpointing — BF16/FP16/FP8 precision strategies, checkpoint design and async saving, resume consistency verification.
4. Loss Spike Debugging — A four-layer (data / numeric / communication / hardware) framework for locating training divergence.

📌 Planned: Slurm heterogeneous scheduling for training jobs, training stability deep-dives.

Reading Path

• New to distributed training: Start with the parallelism overview to build the "partition" mental model, then use the NCCL tuning baselines.
• Cluster operations background: Go straight to NCCL tuning and compare against your own nccl-tests data.
• Shipping LLM training: Read both — parallelism configuration defines topology requirements; network tuning determines whether the configuration can actually run at full speed.

LLM Lab Notes (Hands-on Series)

Based on experiments actually run by the author on GPU clusters (llm_lab/), following the structure "experiment goal → line-by-line code walkthrough → how it works → results → engineering takeaways," breaking down LLM training infrastructure from 0 to 1:

1. 01 · Distributed Training 101: From init_process_group to all_reduce — Process group init, rank/local_rank, and the starting point of collectives.
2. 02 · DDP Minimal Training: Multi-GPU in One Line — DDP parameter broadcast, gradient sync, and the training loop.
3. 03 · Writing Collective Communication Autograd — Wrapping AllReduce in autograd.Function; "communicate in forward, communicate in backward."
4. 04 · Tensor Parallelism 101: Column/Row and the Megatron FFN — Communication cost of the two cuts and the Column→Row no-intermediate-comm trick.
5. 05 · Attention Tensor Parallelism: Splitting QKV by Head — Attention's parallel dividend and its single-communication-point structure.
6. 06 · Building a Megatron-Style Transformer Block from Scratch — LayerNorm/residual/AllReduce autograd assembly + per-segment timing.
7. 07 · Why FlashAttention Is Fast: On-Box Comparison — Tiling, IO-awareness and online softmax, with numbers.

Key Takeaways

• The essence of parallelism is trading communication for memory: if it doesn't fit in memory, partition it; partitioning requires communication; communication needs the network to back it.
• In 10K-GPU training, nine out of ten communication bottlenecks are in the topology, not the link: cross-pod aggregation traffic, ECMP hash imbalance, and PFC deadlock loops.
• Every tuning action must be anchored to a quantifiable baseline: nccl-tests busbw, nsys communication ratio, and training-log MFU.