Symbol beta_link

The passage around this formula

with $\alpha$ the per-step latency and $\beta_{\mathrm{link}}$ the per-link bandwidth. The bandwidth term approaches 2N/$\beta_{\mathrm{link}}$ and stops growing with p ; the latency term grows linearly in p . Small, frequent collectives are therefore latency bound and scale badly, while large ones are bandwidth bound and scale well — which is precisely why gradient bucketing and overlapping communication with backward computation are standard practice rather than optimisations.

Read this part in the article →