Chapter 11 Multi-GPU and Multi-Node Fundamentals

11.1 Interconnect Topologies

11.1.1 PCIe, NVLink, and NVSwitch

Every parallelism strategy you design for a real training run starts with one question: which GPUs can afford to communicate with each other? That question is answered by the interconnect fabric, not your algorithm. Engineers at the mid-to-senior level are expected to know the three layers of intra-node connectivity-PCIe, NVLink, and NVSwitch-and to justify choices like “tensor parallel up to 8” or “allreduce is fast enough here” by pointing at concrete bandwidth numbers. If you answer that question with a vague reference to “fast GPU networking,” you have skipped the most performance-critical part of the system.

Three interconnect technologies govern how GPUs talk to each other inside a single machine, and they sit in a strict bandwidth hierarchy.

PCIe (Peripheral Component Interconnect Express) is the original path between CPU and GPU. A PCIe Gen 4 x16 slot delivers 32 GB/s unidirectional, or 64 GB/s bidirectional. Gen 5 doubles that to 128 GB/s bidirectional. This sounds reasonable until you note that an H100 reads from its own HBM at 3.35 TB/s-PCIe is roughly 26$\times$ slower than the GPU’s own memory bus. GPU-to-GPU communication bridged through the CPU over PCIe rarely exceeds 30-40 GB/s effective and adds CPU-side synchronization latency.

NVLink is NVIDIA’s direct GPU-to-GPU interconnect, bypassing the CPU entirely. Each NVLink lane runs at far higher bandwidth than a PCIe lane. An H100 SXM5 GPU exposes 18 NVLink 4.0 lanes, delivering 900 GB/s of aggregate bidirectional bandwidth per GPU. An A100 SXM4 offers 600 GB/s per GPU via NVLink 3.0.

NVSwitch is the piece that makes NVLink scale beyond point-to-point links. Without it, NVLink connections are bilateral: GPU 0 might directly link to GPUs 1, 2, and 3, but not GPU 7. NVSwitch is a non-blocking crossbar chip-every pair of GPUs in the node sees the full NVLink bandwidth simultaneously, with no contention. A DGX H100 node ships with four NVSwitch 3.0 chips connecting all eight H100 GPUs, providing a full 900 GB/s all-to-all fabric.

The three levels together form a clear mental model:

PCIe is the CPU highway, NVLink is the GPU express lane, and NVSwitch turns that lane into a non-blocking switchboard so every GPU can talk to every other GPU at full speed simultaneously.

The hierarchy exists because different operations have different bandwidth tolerance. CPU-GPU data transfers (loading a checkpoint, streaming host-side data) are infrequent and can tolerate PCIe latency. But collective operations like allreduce-which fire every training step across the tensor-parallel group-happen dozens of times per second and must complete in single-digit milliseconds to stay off the critical path of compute.

NVSwitch’s non-blocking crossbar is the enabling property for tensor parallelism. A tensor-parallel allreduce requires every GPU to simultaneously send and receive partial activation sums. A point-to-point NVLink ring would serialize transfers and create contention at intermediate hops; NVSwitch eliminates both. All eight GPUs in a node can participate in a single allreduce at near-peak wire bandwidth.

The alternative before NVSwitch was a hybrid cube-mesh NVLink topology where bandwidth between non-directly-connected GPUs was lower and asymmetric. NVSwitch replaced that with a flat, predictable bandwidth guarantee between any two GPUs, making performance analysis far simpler and tensor-parallel scaling far more reliable.

A Worked Example

Consider a 70B-parameter model with hidden dimension $h = 8192$, trained with tensor parallelism across 8 H100 SXM5 GPUs in a single DGX H100 node. Batch size $b = 4$, sequence length $s = 2048$, BF16 precision.

At each transformer layer, the tensor-parallel allreduce synchronizes the output activations. The activation tensor has shape $( b , s , h )$ with size:

A ring allreduce over $n = 8$ GPUs transfers approximately $2 \times^{\frac{n - 1}{n}}$ times the data per GPU, so about $268$ MB.

On NVSwitch at 900 GB/s bidirectional (effective allreduce bandwidth $\approx$ 450 GB/s per GPU after protocol overhead):

On PCIe Gen 5 only (128 GB/s bidirectional, shared across all eight GPUs contending for the same CPU-side bandwidth):

A large matmul in a single 70B transformer layer on an H100 takes roughly 1-3 ms. The NVSwitch-backed allreduce at 0.6 ms sits safely below compute and can be overlapped with the next layer’s matrix multiply. The PCIe-backed allreduce at 17 ms is 5-17$\times$ slower than the compute itself and completely dominates step time. Tensor parallelism becomes impractical.

What You Actually Decide in Practice

• Cap tensor parallelism at 8 on current hardware. Apply tensor parallelism only within a single NVSwitch domain-almost always 8 GPUs on DGX-class nodes. Extending to 16 or 32 requires crossing InfiniBand, where effective allreduce bandwidth collapses by 10-20$\times$. Published benchmarks show a 42% throughput drop at TP=16 and a 65% drop at TP=32 relative to TP=8, on the same number of GPUs. These are not edge cases; they are the expected result of the bandwidth cliff.

• Verify topology before assuming crossbar bandwidth. Not all NVIDIA nodes ship with NVSwitch. RTX consumer cards have no NVLink at all. A100 and H100 PCIe variants have NVLink but no NVSwitchbandwidth between non-directly-connected GPUs is lower and non-uniform. Always run nvidia-smi topo -m before assuming full crossbar performance. Two GPUs shown as connected via “SYS” (systemlevel) instead of “NVL” (NVLink) will communicate at PCIe speeds.

• Budget allreduce cost explicitly. For a full-model allreduce over $P$ parameters in BF16, budget approximately $2 P / B_{\text{eff}}$, where $B_{\text{eff}} \approx 0 . 5 \times B_{\text{peakNVLink}}$. For a 7B-parameter model: $2 \times 14$ GB$/ 450$ GB/s $\approx 62$ ms. This is large enough to matter and must be overlapped with gradient computation in the backward pass rather than serialized after it.

• When PCIe-only machines are all you have. Use data parallelism with large micro-batches to amortize communication over more compute per synchronization. Avoid tensor parallelism entirely-the communication-to-compute ratio is always unfavorable. Pipeline parallelism is permissible if point-to-point activation transfers (not allreduces) can fit within the PCIe budget.

• NVSwitch failure handling. A four-chip NVSwitch configuration has redundancy. If one chip fails, NCCL reroutes over the remaining three, dropping peak crossbar bandwidth by roughly 25%. Effective allreduce bandwidth can fall 20-30%. Monitor NVSwitch health via dcgmi health checks before launching multi-day training runs.

How This Shows Up in Interviews

• “Why do practitioners limit tensor parallelism to 8 GPUs?” A strong answer names NVSwitch as the fast all-to-all fabric that makes layer-level allreduce cheap within a node, explains that crossing to a second node requires InfiniBand with 10-20$\times$ lower bandwidth, and cites the 42% throughput loss at TP=16 as a concrete consequence. Weaker answers say “bandwidth” without quantifying it.

• “Walk me through the bandwidth hierarchy inside a DGX H100 node.” The answer covers three levels: PCIe Gen 5 at 128 GB/s bidirectional for CPU-GPU, NVLink 4.0 at 900 GB/s per GPU for GPU-GPU, and NVSwitch providing the non-blocking all-to-all crossbar. Mention that NVSwitch is what makes the bandwidth non-blocking-without it, GPU pairs that don’t share a direct NVLink lane contend.

• “How would you measure the actual allreduce bandwidth available on a node?” Run NCCL’s all_reduce_perf benchmark sweeping message sizes from 1 MB to 10 GB. Look for two regimes: latency-bound at small messages (below $\sim$100 MB) and bandwidth-bound above. The peak bandwidth-bound number is what you should use for step-time estimates. Senior-level follow-up: “What happens to allreduce performance if one NVSwitch chip in the node fails?” The remaining three chips reroute traffic, effective crossbar bandwidth drops by $\sim$25%, and allreduce latency for large tensors rises proportionally. NCCL falls back to a partial ring and peak throughput may degrade 20-30%-enough to extend a 30-day training run by several days.

Key Takeaways

• PCIe Gen 5 delivers 128 GB/s bidirectional; NVLink 4.0 delivers 900 GB/s per GPU; NVSwitch makes that bandwidth non-blocking across all 8 GPUs in a node.

• Tensor parallelism is practical only within a single NVSwitch domain. Crossing InfiniBand for TP increases allreduce cost by 10-20$\times$ and kills throughput.

• Always verify node topology with nvidia-smi topo -m. PCIe-only nodes and NVLink-without-NVSwitch nodes have non-uniform bandwidth that breaks the flat-bandwidth assumption.

• Allreduce cost is approximately $2 P / ( 0 . 5 \times B_{\text{NVLink}} )$; for a 7B model on NVSwitch, this is $\sim$62 ms-significant but overlappable with compute in a correctly pipelined backward pass.

• When NVSwitch hardware is unavailable, drop tensor parallelism from the strategy and rely on data parallelism with large batches or pipeline parallelism with point-to-point sends.

11.1.2 GPU Clusters vs. TPU Pods

The moment your model requires more than one node, the question stops being “which interconnect?” and starts being “what network topology?” The answer determines which parallelism strategies remain efficient and, critically, where you will hit a wall as you scale. Senior engineers at frontier labs are expected to explain why a training run switches from tensor parallelism to pipeline parallelism above a certain node count, and why Google’s TPU-based runs can sustain tensor parallelism at larger group sizes than comparable GPU clusters. The GPU fat-tree cluster and the TPU toroidal mesh are the two dominant answers to this topology question, and they make starkly different trade-offs.

GPU clusters use a fat-tree or dragonfly network topology. Each node-typically 8 GPUs connected by NVSwitch-attaches to a Top-of-Rack (ToR) switch via InfiniBand. Multiple ToR switches connect upward to leaf switches, and leaf switches connect to spine switches. Within a rack of roughly 32 nodes (256 GPUs), any two nodes can reach each other in at most two switch hops, giving all-to-all communication at near-wire bandwidth. Beyond approximately 256 GPUs spanning multiple racks, traffic traverses spine switches shared by many racks. Those uplinks are often the oversubscribed bottleneck, and effective per-node bandwidth for large collectives can drop by 50-80%.

TPU pods use a multi-dimensional toroidal mesh. In a v4 TPU pod, each chip connects directly to six immediate neighbors in a 3D torus via Inter-Chip Interconnect (ICI) at approximately 340 GB/s bidirectional per chip. The topology is physically uniform-there is no hierarchy, no “within-rack” vs. “cross-rack” distinction, and no bandwidth cliff as you scale beyond 256 chips. Allreduce is implemented as a sequence of ring operations along each torus axis, and effective collective bandwidth approaches wire bandwidth regardless of pod size. The constraint is the opposite: communication between distant non-adjacent chips requires multi-hop routing through neighbors, and irregular all-to-all patterns (such as load-imbalanced expert routing) pay a higher penalty than on a fat-tree.

GPU fat-tree clusters give any-to-any communication at full speed up to a topology-dependent threshold, then degrade; TPU toroidal meshes provide uniform, topology-aware bandwidth that scales gracefully but rewards algorithms aligned with nearest-neighbor communication patterns.

For GPU fat-tree clusters, the strength is communication flexibility. Expert parallelism with dynamic token routing, irregular pipeline schedules, and ad-hoc all-to-all operations all work without topology-specific optimization because any two GPUs can communicate directly. The weakness is the bandwidth cliff. When collective operations cross spine switches under 2:1 or 4:1 oversubscription, the effective allreduce bandwidth halves or quarters. This is why standard GPU training recipes switch from tensor parallelism (which demands high-bandwidth allreduce every layer) to pipeline parallelism (which requires only point-to-point activation transfers between adjacent stages) as runs scale across multiple racks.

For TPU pods, the strength is bandwidth uniformity. An allreduce decomposed into per-axis ring operations achieves near-wire bandwidth across the entire pod-the same effective rate at 64 chips and at 4096 chips. This is the hardware reason TPU-based training can sustain tensor-parallel groups of 16 or more chips without the throughput collapse seen on GPU clusters. The weakness is pattern sensitivity: the torus topology is optimized for structured, axis-aligned communication. Expert parallelism with uneven loads or arbitrary point-to-point sends require multi-hop routing that lowers effective bandwidth relative to what a fully non-blocking GPU fat-tree would provide for the same operation.

A Worked Example

Consider a 70B-parameter model in BF16 trained across 1,024 H100 GPUs organized as 128 nodes of 8 GPUs each, with tensor parallelism degree $t = 8$ within each node and data parallelism across all 128 nodes.

Within-rack data-parallel allreduce (32 nodes, 256 GPUs, single fat-tree rack):

The gradient allreduce must synchronize $14$ GB of BF16 parameters across 32 nodes. Each node must transmit approximately $14$ GB $\times \frac{31}{32} \approx 13 . 6$ GB. Assume each node has two NDR InfiniBand ports at 50 GB/s each, giving 100 GB/s bidirectional per node:

At H100 with 50% MFU on a 70B model, a forward-backward pass takes roughly 600-900 ms. The allreduce at 136 ms is $\sim$15-23% of step time-significant but maskable with overlapped gradient communication.

Cross-rack allreduce (128 nodes spanning 4 racks, 2:1 oversubscribed spine):

Effective per-node bandwidth halves to 50 GB/s due to spine oversubscription:

Now allreduce is 30-45% of step time and cannot be fully hidden by compute overlap. The training run is communication-bound. The mitigation is to switch data parallelism to pipeline parallelism across racks, reducing cross-rack traffic to per-microbatch activation sends of $\sim$200 MB at most.

Same run on a TPU v4 pod with 1,024 chips and 340 GB/s ICI per chip:

This is 4-7$\times$ faster than the within-rack GPU case and does not degrade as the pod grows. Tensor parallelism at degree 16 or 32 remains feasible because the per-chip ICI bandwidth is uniformly available at any scale.

What You Actually Decide in Practice

• The 256-GPU threshold on GPU clusters. Within a single non-oversubscribed fat-tree rack of $\sim$256 H100s, you can sustain tensor plus data parallelism without pipeline overhead. Beyond 256 GPUs spanning multiple racks, add pipeline parallelism to reduce cross-rack traffic from full-model allreduces to per-stage point-to-point sends. The rule is not 256 exactly-it is the boundary at which your fabric becomes oversubscribed-but 256 is the practical default for a single rack of NVIDIA DGX nodes.

• Always measure fabric oversubscription before designing the parallelism strategy. Run NCCL’s all_reduce_perf across $N$, $2 N$, and $4 N$ nodes. A bandwidth drop larger than 15% when doubling node count signals oversubscription at the next switch level. Design your cross-node parallelism to minimize volume at that boundary.

• InfiniBand port count per node drives cross-node bandwidth. A node with a single HDR port at 25 GB/s has a 36$\times$ bandwidth asymmetry with its NVSwitch fabric (900 GB/s). A node with four NDR ports at 50 GB/s each gets 200 GB/s cross-node, a 4.5$\times$ asymmetry. The number of InfiniBand ports per node is often the single most important number for estimating cross-node allreduce feasibility.

• Pipeline parallelism is the cross-rack workaround on GPU clusters. Point-to-point activation sends in pipeline parallelism scale as $b \times s \times h$ bytes per microbatch-roughly 200 MB to 1 GB for typical configurations-far smaller than the 14 GB allreduce for a 7B model. Pipeline parallelism tolerates the higher latency of cross-rack InfiniBand because it is sequential and easily overlapped with microbatch compute.

• Designing for TPU topology. When partitioning a model across a TPU torus, map the highest-bandwidth collective (typically tensor-parallel allreduce) to a single torus axis, and map data parallelism to another axis. Mixed-axis communication requires multi-hop routing and lowers effective bandwidth. Misaligned layouts can turn a theoretically 340 GB/s fabric into a 100 GB/s effective one for your specific collective pattern.

How This Shows Up in Interviews

• “Why does the standard recipe use tensor parallelism within a node and pipeline parallelism across nodes?” A complete answer: tensor parallelism requires allreduce every layer; NVSwitch provides 900 GB/s all-to-all within a node; InfiniBand provides 25-100 GB/s between nodes. The 10-36$\times$ bandwidth gap means cross-node tensor parallelism is almost always bandwidth-bound. Pipeline parallelism substitutes point-to-point activation sends for allreduces, dramatically reducing cross-node traffic volume.

• “How does Google sustain larger tensor-parallel groups on TPUs than is typical on GPU clusters?” The answer is ICI topology: each TPU chip has $\sim$340 GB/s to its neighbors uniformly, with no spine-switch bottleneck. Tensor-parallel allreduce implemented as torus rings approaches wire bandwidth at any pod size. On GPU clusters, crossing the rack boundary with allreduce introduces a 2-4$\times$ bandwidth penalty that makes large tensor-parallel groups impractical.

• “You have a 4,096-GPU budget. Walk me through your parallelism assignment.” A strong answer: (1) set $t = 8$ tensor parallel within each NVSwitch node; (2) assess whether 4,096 / 8 = 512 nodes fit in a single non-oversubscribed fabric-probably not, so add pipeline parallelism across racks with enough microbatches to keep bubble fraction below 5%; (3) fill remaining capacity with data parallelism up to the batch size limit. Senior follow-up: “Your InfiniBand fabric is 4:1 oversubscribed at the spine. What changes?” Effective cross-rack bandwidth drops to 25% of peak. Increase pipeline microbatch count to keep bubble fraction acceptable, and reduce data-parallel node count to decrease spine traffic. Consider whether model sharding via ZeRO stage 3 can consolidate the run onto fewer racks to stay within a single non-oversubscribed fat-tree domain.

Key Takeaways

• GPU fat-tree clusters provide any-to-any full-bandwidth communication up to $\sim$256 GPUs within a rack; beyond that, spine-switch oversubscription cuts effective bandwidth by 50-80% for collective operations.

• TPU toroidal meshes deliver uniform $\sim$340 GB/s ICI per chip with no topology cliff, making tensor-parallel allreduce feasible at large group sizes-but irregular all-to-all patterns pay multi-hop penalties.

• The 256-GPU rack boundary is where pipeline parallelism typically enters the GPU cluster strategy, replacing full-model allreduces with small point-to-point activation sends.

• InfiniBand bandwidth per node (25-100 GB/s depending on port count) is 10-36$\times$ lower than NVSwitch bandwidth (900 GB/s); this is the defining asymmetry for all cross-node parallelism decisions.

• Measure actual fabric oversubscription before finalizing any parallelism plan. Published peak bandwidth and realized collective bandwidth routinely differ by 2-4$\times$ on production clusters.

11.2 Collective Communication Primitives

11.2.1 AllReduce, ReduceScatter, and AllGather

When a system design interviewer asks how you synchronize gradients across 64 GPUs, naming “AllReduce” is the floor, not the ceiling. The follow-up-what does it cost, which ring algorithm is used, when would you choose ReduceScatter instead, and how does NVLink bandwidth differ from InfiniBand-is where mid-level and senior answers diverge. Collective communication primitives are the atomic operations from which every parallelism strategy in distributed training is built, and interviewers at frontier labs expect you to reason about them in terms of bytes transferred and wall-clock latency, not just semantics.

A collective operation is any communication pattern in which all $N$ participating processes (called ranks, indexed $0$ through $N - 1$, with the full set called the process group) must cooperate to complete. The six primitives you need are:

• Broadcast: Rank $r$ sends an identical $P$-byte tensor to every other rank. All ranks end with the same tensor. Total bytes transferred $\approx P$.

• Scatter: Rank $r$ splits a tensor of total size $P$ into $N$ equal chunks and sends chunk $i$ to $\text{rank} i$. Total bytes transferred $\approx P$.

• Gather: Each $\text{rank} i$ holds a $P / N$-byte chunk; $\text{rank} r$ collects all $N$ chunks into one $P$-byte tensor. Total bytes transferred $\approx P$.

• Reduce: Same as Gather, but $\text{rank} r$ applies a commutative associative operation (sum, max, min) across corresponding positions as chunks arrive. Total bytes transferred $\approx P$ to the root.

• AllGather: Each rank holds $P / N$ bytes; after the operation, every rank holds all $P$ bytes concatenated. Using a bandwidth-optimal ring, each rank sends $( N - 1 ) / N \times P$ bytes.

• ReduceScatter: Each rank holds a full $P$-byte tensor; after the operation, $\text{rank} i$ holds only the reduction (e.g., sum) of the $i$-th slice across all ranks. Each rank ends with $P / N$ bytes. Per-rank cost $\approx ( N - 1 ) / N \times P \approx P$ for large $N$.

AllReduce is the operation every ML engineer names first. Every rank starts with $P$ bytes, and every rank ends with the fully-reduced $P$-byte result. In the bandwidth-optimal ring implementation the per-rank communication cost is:

The factor of 2 arises because data must first circulate to be accumulated (a ReduceScatter phase) and then circulate again to be distributed to all ranks (an AllGather phase). This $2 P$ bound is tight: no implementation can do better in the bandwidth-limited regime.

AllReduce, ReduceScatter, and AllGather are the three primitives from which every distributed training communication schedule is constructed; the costs $2 P$, $P$, and $P$ per rank should be immediate.

These primitives exist because point-to-point communication is topology-dependent and error-prone at scale. Writing explicit send/receive pairs for a ring of 64 GPUs produces brittle code; a single all_reduce call lets NCCL choose the optimal topology-aware algorithm and pipeline multiple operations over available links simultaneously.

The ring algorithm achieves near-perfect link saturation regardless of world size. At $N = 8$ the per-rank cost is $7 / 8 \times 2 P = 87 . 5 \%$ of the asymptotic $2 P$. At $N = 64$ it is $98 . 4 \%$. Compare this to a naive star topologyone root receives from all others then broadcasts-which creates a $2 ( N - 1 ) P$ bottleneck at the root and scales linearly with world size.

ReduceScatter and AllGather are individually half the cost of AllReduce because each moves only $P$ bytes per rank instead of $2 P$. This asymmetry is the economic argument behind ZeRO-style optimizer sharding: decompose AllReduce into ReduceScatter (leave partial results distributed) plus AllGather (reconstitute them), and insert optimizer computation in the gap without spending any extra bandwidth.

A Worked Example

Consider a data-parallel setup with 8 H100s on a single NVLink node, training a 7B-parameter model in BF16. The gradient tensor is $7 \times 10^{9} \times 2$ bytes $= 14$ GB.

AllReduce cost:

H100 NVLink 4 provides 900 GB/s aggregate bidirectional bandwidth per GPU. In a ring, each link is shared across concurrent steps, so effective throughput is roughly half the aggregate per direction. Benchmarks on an 8-GPU node typically yield $\sim$277 GB/s for large-tensor AllReduce:

ReduceScatter only: Each rank ends with a $14 / 8 = 1 . 75$ GB shard of fully-summed gradients. Per-rank cost is half:

If you take an Adam step on the 1.75 GB shard during this window (roughly 3-5 ms of compute on an H100) and then AllGather (another $\approx$44 ms), total wall time is $\approx$88-90 ms - identical to AllReduce. The optimizer work was absorbed for free.

Cross-node comparison: The same 7B gradient sync across 64 GPUs on two InfiniBand HDR nodes (200 GB/s per link):

A $5 \times$ slowdown versus NVLink. This is a concrete reason why tensor parallelism (which issues AllReduce every layer) is confined to the intra-node NVLink domain, while data parallelism (which issues AllReduce once per gradient-sync cycle) tolerates the slower inter-node link.

What You Actually Decide in Practice

• Use AllReduce for gradient sync when each rank holds a full copy of the model. Call dist.all_reduce(param.grad, op=dist.ReduceOp.SUM) after the backward pass. Divide by world size in the optimizer, not in the communication call, to keep the two concerns separate.

• Switch to ReduceScatter + AllGather whenever optimizer memory is the bottleneck. The bandwidth cost is identical to AllReduce, so there is no reason to use plain AllReduce once you have the ZeRO-1 infrastructure in place.

• Profile before assuming AllReduce is the bottleneck. On an 8-GPU NVLink node, 88 ms of gradient sync against 500 ms of forward+backward compute for a 7B model is under 15% overhead. Across nodes at 138 ms, it may dominate and justify tensor parallelism or gradient compression.

• Use async_op=True to overlap AllReduce with the optimizer step on the next micro-batch. NCCL executes asynchronously on the GPU; calling all_reduce before optimizer.step() lets communication and compute pipeline if they target different parameter groups.

• Do not confuse AllGather with Broadcast. AllGather assembles distinct per-rank chunks into one concatenated tensor; Broadcast replicates the same tensor from one sender. Using Broadcast where AllGather is needed silently produces wrong results and is a common copy-paste bug when building FSDP-style code from scratch.

How This Shows Up in Interviews

• “Walk me through how gradients are synchronized in a standard data-parallel training run over 8 GPUs.” Strong answer: each GPU computes gradients on its local mini-batch via backward pass; then calls AllReduce (ring algorithm, sum) to produce the globally averaged gradient; then divides by world size (or uses ReduceOp.AVG); then each GPU takes an identical optimizer step. Cost: $2 P$ bytes per rank.

• “What is the communication overhead of AllReduce for a 70B-parameter model in BF16?” Strong answer: $70 B \times 2$ bytes $= 140$ GB; AllReduce per-rank cost $\approx 280$ GB; at 200 GB/s inter-node InfiniBand, roughly 1.4 seconds per step-likely unacceptable without gradient compression, pipeline parallelism, or switching to ReduceScatter-based ZeRO.

• Senior follow-up: “NCCL reports 277 GB/s on your 8-GPU NVLink node, but the H100 spec sheet says 900 GB/s. Where did the bandwidth go?” Strong answer: three factors. First, 900 GB/s is aggregate bidirectional; the ring uses each link in both directions simultaneously, but the peak per-direction rate is lower. Second, ring AllReduce parallelizes across all links, but each GPU’s effective slice of the bandwidth is shared across $N - 1$ concurrent ring steps. Third, small tensor sizes do not saturate the link because latency overhead dominates; benchmarked throughput rises sharply with tensor size and plateaus around several hundred MB.

Key Takeaways

• AllReduce leaves every rank with the full reduced result; per-rank cost $\approx 2 P$ bytes.

• ReduceScatter leaves each rank with one reduced shard of size $P / N$; per-rank cost $\approx P$ bytes.

• AllGather assembles per-rank shards so every rank holds the full $P$-byte tensor; per-rank cost $\approx P$ bytes.

• On NVLink (benchmarked $\sim$277 GB/s effective), a 14 GB gradient AllReduce costs $\sim$88 ms; on InfiniBand (200 GB/s) it costs $\sim$138 ms-topology determines whether communication is negligible or the dominant cost.

• Prefer one large fused AllReduce over many small per-parameter AllReduces; NCCL kernel launch overhead is non-trivial for tensors under $\sim$10 MB.

11.2.2 Why AllReduce Equals ReduceScatter Plus AllGather

The identity AllReduce $=$ ReduceScatter $+$ AllGather is the single algebraic fact that makes ZeRO’s “zero overhead” claim true. When an interviewer asks why sharding the optimizer state is free in the bandwidth-limited regime, this equivalence is the complete answer. It surfaces in senior-level system design questions and-more importantly-it is the reasoning you need to correctly evaluate any claim that a particular communication schedule is bandwidth-optimal. Missing it means you will accept hand-wavy performance numbers without being able to verify them on a whiteboard.

In the ring algorithm, AllReduce already executes as two sequential phases internally:

1. ReduceScatter phase. The tensor of size $P$ is logically split into $N$ equal slices of size $P / N$. Over $N - 1$ ring steps, each rank passes one slice clockwise and accumulates the incoming slice with its local copy. After $N - 1$ steps, $\text{rank} i$ holds the fully-reduced (summed) value of slice $i$ across all $N$ ranks-and nothing else. Per-rank bytes transmitted: $( N - 1 ) / N \times P$.

2. AllGather phase. Over another $N - 1$ ring steps, each rank propagates its fully-reduced slice to the rest of the ring. After $N - 1$ steps, every rank holds all $N$ fully-reduced slices-the complete AllReduce result. Per-rank bytes transmitted: another $( N - 1 ) / N \times P$.

Total per-rank cost:

Because both $P$-byte paths happen to appear as two sub-operations, you can break AllReduce at the boundary between phases and do arbitrary computation in between:

as long as $f$ is applied element-wise within each rank’s shard and does not change the shard size. Optimizer steps satisfy this exactly: $\text{rank} i$ applies Adam to its $P / N$ gradient shard and produces a $P / N$ updated parameter shard, which AllGather then reconstitutes.

Any computation that operates independently on each rank’s shard and produces a same-sized shard can be inserted between ReduceScatter and AllGather at zero additional bandwidth cost.

The naive mental model for data-parallel gradient sync is: run backward $\rightarrow$ AllReduce $\rightarrow$ optimizer step. The implicit assumption is that every GPU needs the full reduced gradient before updating any parameter. That assumption is wrong, and the decomposition exposes why.

After ReduceScatter, $\text{rank} i$ holds the exact, fully-summed gradients for its assigned $P / N$ parameters. That is sufficient information to run a complete Adam step on those parameters-it does not need the gradients for any other parameter. The AllGather is required only to redistribute the updated parameter values so all ranks agree on the full model before the next forward pass.

Because the total bytes transferred equals the AllReduce baseline ($2 P$), there is no bandwidth penalty. The optimizer state-first moment, second moment, master weights-can now be sharded: $\text{rank} i$ only ever needs the optimizer state for its $P / N$ slice. On a 7B model in BF16, naive DDP carries $16 \times P$ bytes of optimizer state per rank (full Adam in mixed precision); ZeRO stage 1 reduces that by a factor of $N$. For $N = 64$, that is a $64 \times$ reduction in optimizer memory at zero communication cost.

The contrast with the obvious alternative-gather all gradients to one rank, run the full optimizer, broadcast-is stark. That approach costs $( N - 1 ) P$ to gather and $( N - 1 ) P$ to broadcast, totalling $2 ( N - 1 ) P$, which is $N$ times worse than the ring for large $N$. More importantly, it also creates an $N \times$ memory bottleneck on the root rank. The ring decomposition eliminates both problems.

A Worked Example

Four ranks, each holding a gradient vector of length 4: [a,b,c,d], [e,f,g,h], [i,j,k,l], [m,n,o,p]. The desired AllReduce result is position-wise sum: $[ a + e + i + m , b + f + j + n , c + g + k + o , d + h + l + p ]$.

After ReduceScatter:

• Rank 0: $a + e + i + m$ (slice 0, fully summed)

• Rank 1: $b + f + j + n$ (slice 1, fully summed)

• Rank 2: $c + g + k + o$ (slice 2, fully summed)

• Rank 3: $d + h + l + p$ (slice 3, fully summed)

Optimizer step (between phases): Each rank applies its Adam update to its scalar. Say rank 0’s updated parameter for position 0 is $\theta_{0}^{\prime}$; rank 1’s for position 1 is $\theta_{1}^{\prime}$; and so on.

After AllGather: Every rank holds $[ \theta_{0}^{\prime} , \theta_{1}^{\prime} , \theta_{2}^{\prime} , \theta_{3}^{\prime} ]$ - a fully synchronized, fully updated parameter tensor.

Total bytes per rank: one scalar sent in ReduceScatter (to 3 neighbors over 3 steps), one scalar sent in AllGather (to 3 neighbors over 3 steps). Exactly $2 \times ( N - 1 ) / N \times P = 2 \times 3 / 4 \times 4 = 6$ scalar-sends per rank, versus 6 for a ring AllReduce. Identical.

At production scale: a 7B model in BF16 gives $P = 14$ GB. In ZeRO stage 1 across 8 ranks, ReduceScatter delivers 1.75 GB shards. An Adam step on 1.75 GB of parameters takes $\sim$3 ms on an H100 (gradient application is memory-bandwidth bound at roughly 900 GB/s HBM for small kernels, plus CUDA overhead). AllGather then reconstitutes the 14 GB tensor. The Adam step is effectively free: it runs during the $\sim$44 ms ReduceScatter ring, not in series with communication.

What You Actually Decide in Practice

• ZeRO stage 1 is always worth enabling. The bandwidth is identical to naive DDP AllReduce. The only implementation cost is maintaining per-rank optimizer state slices and calling dist.reduce_scatter and dist.all_gather instead of dist.all_reduce. PyTorch’s FSDP automates this; if you are rolling your own, the two-call sequence is five lines of code.

• ZeRO stage 3 is not free: it costs $3 P$ per rank, not $2 P$. The extra $P$ is a forward-pass AllGather needed to materialize each layer’s sharded parameters before the forward computation. This has no counterpart in ZeRO stage 1, where parameters remain replicated. Measure the 50% communication increase against your memory savings before committing to stage 3.

• The optimizer step between ReduceScatter and AllGather is a hard serial dependency. AllGather cannot begin until every rank has finished its Adam step on its shard. If your optimizer has pathologically slow kernels (e.g., due to BF16-FP32 casting loops), it adds directly to latency. Fuse or kernel-optimize the optimizer step if it exceeds $\sim$5 ms per layer.

• Verify your ReduceScatter + AllGather pair uses the same ring topology as your AllReduce. NCCL selects algorithms based on message size and world size; if you call the two operations separately, NCCL may choose a tree algorithm for one and a ring for the other, breaking the bandwidth-equivalence assumption. Force the ring algorithm via environment variable NCCL_ALGO=Ring when benchmarking.

• The equivalence holds only in the bandwidth-limited regime. For small tensors ($< \sim 1$ MB), the latency of two ring passes exceeds a single fused AllReduce kernel. Do not decompose gradient sync for the embedding layer or small classification heads; the latency overhead will exceed the memory savings.

How This Shows Up in Interviews

• “Why is ZeRO stage 1 claimed to have zero communication overhead compared to DDP?” Strong answer: AllReduce already decomposes into ReduceScatter $+$ AllGather internally. ZeRO stage 1 just inserts the per-rank optimizer step between those two phases. The bandwidth is $2 P$ in both cases; the optimizer state is sharded at no additional cost.

• “What is the communication cost of ZeRO stage 3 compared to ZeRO stage 1?” Strong answer: ZeRO stage 3 costs $3 P$ per rank versus $2 P$ for ZeRO stage 1. The extra $P$ is a per-layer parameter AllGather in the forward pass that does not exist in ZeRO stage 1 (where parameters remain replicated). The tradeoff is $1 . 5 \times$ more communication in exchange for $N \times$ parameter memory savings.

• Senior follow-up: “Measured throughput on an 8-GPU FSDP run is 25% lower than DDP for the same 7B model. You know ZeRO 3 should cost $3 P$ versus $2 P$ = 50% more communication. Why is the throughput hit only 25%?” Strong answer: three reasons. First, communication and computation overlap significantly: FSDP prefetches the next layer’s AllGather while the GPU is computing the current layer’s forward, so the extra $P$ is partly hidden. Second, FSDP enables larger effective batch sizes per GPU by freeing optimizer memory, which improves GPU utilization and partially offsets the communication overhead. Third, the $3 P$ versus $2 P$ comparison assumes communication is 100% of wall time; in practice, the forward and backward flops dominate and the relative communication fraction is smaller than the raw byte ratio implies.

Key Takeaways

• AllReduce $=$ ReduceScatter $+$ AllGather: total bandwidth is $2 P$ per rank in both cases; the decomposition is bandwidth-neutral.

• ZeRO stage 1 inserts the optimizer step into the gap that already exists between ReduceScatter and AllGather, achieving $N \times$ optimizer memory reduction at zero bandwidth cost.

• ZeRO stage 3 adds a third $P$-byte AllGather per layer in the forward pass, raising total communication to $3 P$ per rank-not free, but often worth it for very large models.

• The equivalence is bandwidth-based; for small tensors in the latency-dominated regime, a fused AllReduce kernel beats two separate ring passes.

• The key invariant: after ReduceScatter, each rank owns a fully-summed gradient shard; after AllGather, each rank owns the fully-updated parameter tensor.

11.3 The Software Stack

11.3.1 NCCL and torch.distributed

An interviewer at a frontier lab will frequently ask you to trace a single dist.all_reduce() call from your Python training loop all the way down to the NVLink packets leaving the GPU. This question appears at the senior level because it separates engineers who have used torch.distributed as a black box from those who understand the contract each layer provides. Getting this right matters in production: a misunderstood backend choice causes either correctness failures (wrong gradients silently) or catastrophic hangs that waste hours of cluster time.

The distributed communication stack for GPU training consists of two layers sitting between your Python gradient tensors and the NVLink wires. The bottom layer is NVIDIA’s Collective Communications Library (NCCL), a closed-source C library that owns the problem of routing multi-GPU collective operations as efficiently as possible on NVIDIA hardware. The top layer is torch.distributed, a Python API that wraps NCCL-and alternative backends-behind a single, hardware-agnostic interface.

NCCL’s job starts at process-group initialization. When you call dist.init_process_group(backend="nccl"), NCCL runs a topology discovery phase: it queries which GPUs are connected via NVLink, which are connected only via PCIe, and which sit on remote nodes reachable over InfiniBand or Ethernet. From that topology it precomputes optimal reduction trees and ring schedules for every collective. Critically, the rendezvous at init_process_group is a lightweight coordination step-it uses a small TCP or shared-filesystem handshake just so ranks can find each other. No tensor data ever flows through that channel. All actual payload moves via NCCL kernels launched directly on the GPU, bypassing the host CPU entirely once the operation is underway.

When your code reaches dist.all_reduce(tensor, op=dist.ReduceOp.SUM), the CPU thread calls into NCCL through torch.distributed’s nccl backend shim. NCCL enqueues a CUDA kernel on the device stream; that kernel manages point-to-point NVLink transfers between peers, performs the partial reductions on-chip, and writes the final result back into tensor in place on every rank-all without the CPU being involved in the data movement.

The second role of torch.distributed is backend portability. The same collective API accepts backend="gloo" to run on CPU using Ethernet, or backend="mpi" for MPI-based clusters. For debugging on a laptop, switching to gloo lets you verify correctness of your distributed logic without needing a GPU. Performance will be dramatically lower, but logical correctness transfers cleanly.

The contract is: NCCL owns topology-aware routing and kernel efficiency; torch.distributed owns the Python API and backend portability-you operate at the torch.distributed level and trust NCCL to saturate your NVLink bandwidth.

The alternative to this stack-manual point-to-point sends and receives through a general-purpose network library-requires every programmer to reason about reduction trees, ring topologies, and hardware-specific bandwidth asymmetries. NCCL abstracts all of that. Its topology-aware ring-allreduce or recursive halving-doubling schedules extract close to peak NVLink bandwidth without requiring the caller to know whether the GPUs are on the same NVLink switch or spread across two nodes.

The key trade-off that torch.distributed makes is that it couples correctness to strict collective-call symmetry. Every rank in the process group must execute the same collective operations in the same order. If one rank diverges-say, a conditional branch that skips an all_reduce on rank 3 while ranks 0-2 are waitingall the other ranks block indefinitely. This synchronous-collective model simplifies reasoning about gradient consistency but is an operational hazard: a one-line Python exception on a single rank will hang the entire job.

Before NVLink-based stacks existed, multi-GPU communication had to stage data through host memory, copy it across PCIe to the CPU, retransmit over Ethernet to another node’s CPU, copy back over PCIe, and finally land it on the target GPU. NCCL on NVLink-equipped nodes eliminates every intermediate hop for intra-node communication. The bandwidth difference is roughly an order of magnitude: PCIe gen 4 delivers $\sim$64 GB/s bidirectional per slot; H100 NVLink 4 delivers $\sim$900 GB/s aggregate across all 18 links.

A Worked Example

Consider a data-parallel training job: 4 H100s on one node, each holding a copy of a model whose gradient tensor is 400 MB (100 million float32 parameters, $100 \times 10^{6} \times 4 = 400$ MB). After the backward pass, each rank calls:

dist.all_reduce(grad_tensor, op=dist.ReduceOp.SUM)

Step 1: NCCL selects a ring-allreduce schedule over 4 ranks. In a ring-allreduce, the operation proceeds in two phases-reduce-scatter then all-gather-each requiring $( N - 1 )$ steps of message passing around the ring. Step 2: Bytes transferred per rank. Each rank sends and receives data equal to:

where $S = 400$ MB and $N = 4$:

The factor of 2 accounts for the reduce-scatter phase (data flows in) and the all-gather phase (result flows out). Step 3: Expected wall-clock time at effective NVLink bandwidth of 300 GB/s (a realistic single-link utilization on a 4-GPU node, not the full 900 GB/s aggregate):

In practice, benchmarking this exact configuration on H100 yields measured effective bandwidth in the range of 250-300 GB/s, translating to 2-2.5 ms for the full allreduce. This is the number you hold in your head when sizing whether gradient sync dominates your step time.

What You Actually Decide in Practice

• Always use nccl for GPU training; use gloo only for CPU debugging. gloo over Ethernet is 10-50$\times$ slower than nccl over NVLink and will make your job communication-bound even at small scale. Never accidentally ship gloo to a GPU cluster.

• Treat collective calls as global barriers. Every rank must call the same collective in the same order. Structure your code so conditionals never cause a rank to skip or add a collective. A common bug: wrapping a logging all_gather inside if rank == 0 on only one rank-this will hang the job.

• Use async_op=True to overlap communication with computation. dist.all_reduce(tensor, async_op=True) returns a handle immediately; the CPU can dispatch the next backward layer while NCCL drains the previous gradient bucket over the wire. DDP’s gradient bucketing relies on exactly this pattern.

• Set NCCL_DEBUG=INFO when diagnosing hangs or low bandwidth. NCCL will log its topology discovery, selected algorithm, and ring schedule. If you see NCCL picking a PCIe path instead of NVLink, the GPU placement on the node is wrong.

• For cross-node jobs, InfiniBand backend matters as much as NVLink. NCCL supports GPUDirect RDMA to write directly from GPU memory to InfiniBand without staging through host RAM. Without it, cross-node bandwidth drops by 2-3$\times$. Verify RDMA is enabled with nccl_p2p_disable=0 and confirm with NCCL_DEBUG=TRACE.

How This Shows Up in Interviews

• “Walk me through what happens when your training loop calls dist.all_reduce() on a gradient tensor.” A strong answer traces the path: Python API $\rightarrow$ torch.distributed nccl shim $\rightarrow$ NCCL enqueues a CUDA kernel $\rightarrow$ GPU executes ring-allreduce over NVLink $\rightarrow$ result written back in-place on all ranks. Mention that the CPU is not in the data path after kernel launch.

• “Why might you choose gloo over nccl?” The one valid reason: running on CPU, typically for debugging or CI environments without GPUs. On a GPU node, gloo is never the right production choice.

• “If rank 2 throws a Python exception mid-training, what happens to the other 7 ranks?” They hang at the next collective call waiting for rank 2 to arrive. Recovery requires catching the exception on rank 2, broadcasting an error signal to all ranks, and having everyone call dist.destroy_process_group() cleanly-or just letting the job manager detect the dead process and restart.

• Senior follow-up: “How does DDP overlap gradient communication with the backward pass, and what are the limits of that overlap?” DDP partitions parameters into buckets (default 25 MB). When the backward hook fires for the last parameter in a bucket, it immediately launches an async all_reduce for that bucket while the backward pass continues computing earlier-layer gradients. The limit: the first bucket’s allreduce must complete before the optimizer step, so the final bucket on the critical path is not overlapped. For very large models with few parameters per layer, buckets serialize and overlap collapses.

Key Takeaways

• NCCL translates collective operations into topology-optimal CUDA kernels; torch.distributed wraps NCCL behind a portable Python API that also supports gloo for CPU.

• Tensor data never flows through the rendezvous channel; init_process_group is only coordination. All payload moves GPU-to-GPU via NCCL kernels on NVLink or InfiniBand.

• Every rank must execute identical collectives in identical order; a single rank skipping one collective hangs the entire job.

• Use async_op=True to overlap gradient communication with backward computation; this is the foundation of DDP’s gradient bucketing.

• On H100 NVLink 4, expect $\sim$250-300 GB/s effective allreduce bandwidth per rank for large tensors; always benchmark rather than assuming theoretical peak.

11.3.2 Benchmarking Bandwidth in Practice

When an interviewer asks “how long will the gradient allreduce take for a 70B model?” they are really asking two questions at once: do you know the formula for bytes transferred, and do you know that your answer is only meaningful if you have validated it with a real measurement? I have seen senior engineers confidently quote NVLink’s theoretical 900 GB/s peak in response to this question-and have that answer immediately undermined when the interviewer asks why production jobs run at 30% of that number. Knowing how to benchmark collective bandwidth correctly, and knowing what causes the gap between peak and achieved, is a genuine differentiator at the staff level.

Measuring collective bandwidth requires getting three things right: the timing methodology, the synchronization model, and the byte-count formula. Each is independently easy to get wrong.

Timing. GPU operations are asynchronous from the CPU. When you call dist.all_reduce() and immediately read time.time(), you are measuring the time to submit the kernel, not to complete it. The correct idiom is:

torch.cuda.synchronize()
dist.barrier()
t0 = time.time()
dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
torch.cuda.synchronize()
dist.barrier()
t1 = time.time()
elapsed = t1 - t0

torch.cuda.synchronize() blocks the CPU until all pending CUDA kernels on the current device have completed. dist.barrier() ensures all ranks have reached the same point before the clock starts and after it stops, so you are measuring the slowest rank’s time-which is the correct quantity for a collective that requires all participants.

Warm-up. The first execution of an NCCL kernel involves just-in-time compilation, peer-to-peer connection setup, and CUDA context initialization. Run at least one-preferably three-warm-up iterations before recording any timing. Otherwise your first measurement is dominated by one-time overhead that will never appear again in steady-state training.

Byte-count formula. The bytes transferred per rank depends on which collective you are measuring:

where $S$ is the tensor size in bytes per rank and $N$ is the world size. The factor of 2 in allreduce reflects the two phases of the ring algorithm: reduce-scatter (each rank sends chunks out) and all-gather (each rank receives the final result). Reduce-scatter and all-gather each account for half. Because allreduce $=$ reduce-scatter $+$ all-gather, their individual bandwidth numbers add up to the allreduce number-which is a useful sanity check.

Effective bandwidth is the agreed-upon convention for comparing hardware utilization across configurations; it is always measured bytes divided by wall-clock time, computed once synchronization is complete across all ranks.

The alternative-trusting spec sheets-fails for two reasons. First, NVLink’s 900 GB/s aggregate is the sum across all 18 links on an H100; any single rank-to-rank transfer uses a subset. Second, real workloads suffer from contention, buffer setup latency, and algorithmic overhead (e.g., ring-allreduce is not embarrassingly parallel). Benchmarking gives you the actual operating point. When the measured bandwidth is dramatically below peak, you have actionable information: check NCCL topology selection (NCCL_DEBUG=INFO), verify NVLink vs PCIe path, test different tensor sizes to identify the knee in the bandwidth curve.

The explicit warm-up requirement mirrors everything we know about GPU performance measurement from single-device profiling: the first kernel launch is never representative. The same principle that demands warm-up for torch.compile or FlashAttention applies here.

A Worked Example

Setup: 4 H100s on one node, world_size = 4. Tensor: 100 million float32 elements per rank, so $S = 100 \times 10^{6} \times 4 = 400$ MB.

Allreduce measurement:

Suppose the measured elapsed time after synchronization and barrier is $t = 2 . 16$ ms. Then:

Reduce-scatter measurement on the same tensor (input shape: $[ 4 , 100$M$]$, output shape: $[ 100$M$]$):

Suppose measured time is $t = 4 . 3$ ms:

The gap between allreduce (277 GB/s) and reduce-scatter (70 GB/s) at first looks surprising. Allreduce at $2 \times$ the bytes should take $2 \times$ as long, yielding the same bandwidth-yet reduce-scatter runs at roughly one-quarter the rate. The difference likely comes from two sources. First, NVIDIA’s SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) hardware acceleration can perform the reduction inside the NVLink switch fabric for allreduce, effectively halving the traffic seen by the GPUs. Second, NCCL invests more engineering in allreduce tuning since it is the highest-frequency production operation. This is precisely why benchmarking beats arithmetic: the hardware does not always behave as the formula predicts.

What You Actually Decide in Practice

• Always warm up at least 3 iterations before recording. One warm-up is often not enough if NCCL needs to establish peer connections on first use. Three iterations is conservative but reliable.

• Apply both torch.cuda.synchronize() and dist.barrier() around your timer. Synchronize alone measures your own rank’s completion; barrier ensures you capture the slowest rank, which determines real step time.

• Sweep tensor sizes across 1 MB to 10 GB. NCCL has a latency-dominated regime (small tensors, typically below $\sim$1 MB) and a bandwidth-dominated regime. The knee is usually between 10-100 MB. Gradient tensors from large models land firmly in the bandwidth-dominated regime.

• Do not assume reduce-scatter and allreduce achieve the same bandwidth. As the worked example shows, the ratio can be 4$\times$ in favor of allreduce due to in-network compute. If your FSDP or ZeRO plan hinges on reduce-scatter bandwidth, measure it directly.

• Cross-node bandwidth is typically 5-10$\times$ lower than intra-node NVLink. An 8-GPU node connected to another 8-GPU node via InfiniBand HDR (200 Gb/s, $\approx$ 25 GB/s) has a cross-node bottleneck that is 36$\times$ narrower than NVLink per node. Parallelism strategies must respect this asymmetry: pipeline parallelism across nodes, tensor parallelism within nodes.

How This Shows Up in Interviews

• “Estimate how long the gradient sync takes for a 70B parameter model in bf16 across 8 H100s.” Gradient tensor per rank: $70 \times 10^{9} \times 2$ bytes $= 140$ GB. Allreduce bytes per rank: $2 \cdot \frac{7}{8} \cdot 140$ GB $= 245$ GB. At 250 GB/s effective bandwidth: $\approx$ 1 second. A strong answer gives the formula, plugs in realistic bandwidth, and then notes that DDP with gradient bucketing hides much of this behind the backward pass-so raw sync time overstates the wall-clock cost.

• “Why do you need to call torch.cuda.synchronize() before stopping the timer?” Because CUDA kernel dispatch is asynchronous; the CPU returns from dist.all_reduce() before the GPU has finished. Without synchronize, you measure submission latency ($\sim \mu$s), not transfer time ($\sim$ms).

• “You benchmarked allreduce and got 100 GB/s on a 4-GPU H100 node. NVLink peak is 900 GB/s. What do you check?” First, confirm NCCL is using NVLink and not falling back to PCIe (NCCL_DEBUG=INFO). Second, check tensor size-below $\sim$10 MB, latency dominates and bandwidth appears low. Third, verify warm-up was done and synchronization is correct. Fourth, check whether SHARP in-network compute is enabled, which can boost allreduce specifically.

• Senior follow-up: “How would you benchmark whether your pipeline-parallel job is compute-bound or communication-bound across nodes?” Instrument both the forward/backward compute time per microbatch and the inter-node send/recv latency. Compute is bound if the GPU utilization on intermediate ranks stays near 100% while the first and last ranks are idle. Communication is bound if you see the pipeline bubble shrink when you reduce the tensor size sent between stages. Concretely: add torch.cuda.Event timers bracketing the compute and send/recv separately, record them on each rank, and aggregate across steps.

Key Takeaways

• Always warm up and always call torch.cuda.synchronize() plus dist.barrier() around your timer; skipping either gives numbers that are not reproducible and not meaningful.

• The byte-count formula for allreduce is $2 ( N - 1 ) / N \cdot S$; for reduce-scatter and allgather separately, it is $( N - 1 ) / N \cdot S$ each-these add up correctly since allreduce $=$ reduce-scatter $+$ allgather.

• Measured bandwidth for allreduce and reduce-scatter can differ by $3$-$4 \times$ on the same hardware due to in-network SHARP acceleration and differential NCCL optimization; always benchmark both if your architecture uses both.

• Intra-node NVLink bandwidth ($\sim$250-300 GB/s effective per rank on H100) is 10$\times$ higher than typical cross-node InfiniBand; parallelism placement must respect this gap.

• Theoretical peak numbers are starting points for estimates, not commitments; real training jobs routinely achieve 30-60% of advertised NVLink bandwidth for collectives.