multiplication

The passage around this formula

The cost structure of a collective is arithmetic, not policy. For a ring AllReduce over p ranks reducing N bytes at per-link bandwidth B with per-hop latency $\alpha$ , each rank moves $T_{\mathrm{ring}} \approx 2\,\frac{p-1}{p}\cdot\frac{N}{B} + 2\,(p-1)\,\alpha$. so bandwidth cost saturates near 2N/B while the latency term grows linearly in p . Two consequences follow. The slowest link sets the pace for every rank, because the operation does not complete until all ranks have contributed. And large jobs avoid large collectives: Meta reports that multi-dimensional parallelism keeps “the number of GPUs in the largest collective to hundreds of GPUs even when running a job that is tens of thousands of GPUs,” which is why their analysis focuses on…

Read this part in the article →