Symbol α

The passage around this formula

…over p devices and N bytes, the operation decomposes into a reduce-scatter followed by an all-gather, each of p - 1 steps in which every device sends N/p bytes. Total time is approximately $T_{\mathrm{allreduce}} \approx 2\,(p-1)\,\alpha + 2\,\frac{p-1}{p}\cdot\frac{N}{\beta_{\mathrm{link}}}$. with $\alpha$ the per-step latency and $\beta_{\mathrm{link}}$ the per-link bandwidth. The bandwidth term approaches 2N/$\beta_{\mathrm{link}}$ and stops growing with p ; the latency term grows linearly in p . Small, frequent collectives are therefore latency bound and scale badly, while large ones are…

Read this part in the article →