$$T_{\mathrm{allreduce}} \approx 2\,(p-1)\,\alpha + 2\,\frac{p-1}{p}\cdot\frac{N}{\beta_{\mathrm{link}}}$$

Equation 30 · AI Hardware & Semiconductors

What an AI Accelerator Actually Is: Silicon, Packaging, and the Memory It Can Reach

This equation gives an approximation: it relates the quantities while allowing an approximation.

Their cost has a shape worth internalising. For the standard ring formulation of an all-reduce over p devices and N bytes, the operation decomposes into a reduce-scatter followed by an all-gather, each of p - 1 steps in which every device sends N/p bytes. Total time is approximately $T_{\mathrm{allreduce}} \approx 2\,(p-1)\,\alpha + 2\,\frac{p-1}{p}\cdot\frac{N}{\beta_{\mathrm{link}}}$. with $\alpha$ the per-step latency and $\beta_{\mathrm{link}}$ the per-link bandwidth. The bandwidth term approaches 2N/$\beta_{\mathrm{link}}$ and stops growing with p ; the latency term grows linearly in p . Small, frequent collectives are therefore latency bound and scale badly, while large ones are bandwidth bound and scale well — which is precisely why gradient bucketing and overlapping…

Meanings in this article

• $p$: the latency term grows linearly in.
• $\alpha$: the per-step latency and $\beta_{\mathrm{link}}$ the per-link bandwidth.
• $N$: the number of bytes.
• $\beta_{\mathrm{link}}$: the per-link bandwidth.

Equation guide → · Article →