addition

The passage around this formula

Their cost has a shape worth internalising. For the standard ring formulation of an all-reduce over p devices and N bytes, the operation decomposes into a reduce-scatter followed by an all-gather, each of p - 1 steps in which every device sends N/p bytes. Total time is approximately $T_{\mathrm{allreduce}} \approx 2\,(p-1)\,\alpha + 2\,\frac{p-1}{p}\cdot\frac{N}{\beta_{\mathrm{link}}}$. with $\alpha$ the per-step latency and $\beta_{\mathrm{link}}$ the per-link bandwidth. The bandwidth term approaches 2N/$\beta_{\mathrm{link}}$ and stops growing with p ; the latency term grows linearly in p . Small, frequent collectives are therefore latency bound and scale badly, while large ones are bandwidth bound and scale well — which is precisely why gradient bucketing and overlapping…

Read this part in the article →