← Mathematical compendium

Published equation contexts

Tring≈2 p−1p⋅NB+2 (p−1) αT_{\mathrm{ring}} \approx 2\,\frac{p-1}{p}\cdot\frac{N}{B} + 2\,(p-1)\,\alpha

Why this formula appears here

The cost structure of a collective is arithmetic, not policy. For a ring AllReduce over p ranks reducing N bytes at per-link bandwidth B with per-hop latency α\alpha , each rank moves Tring≈2 p−1p⋅NB+2 (p−1) αT_{\mathrm{ring}} \approx 2\,\frac{p-1}{p}\cdot\frac{N}{B} + 2\,(p-1)\,\alpha. so bandwidth cost saturates near 2N/B while the latency term grows linearly in p . Two consequences follow. The slowest link sets the pace for every rank, because the operation does not complete until all ranks have contributed. And large jobs avoid large collectives: Meta reports that multi-dimensional parallelism keeps “the number of GPUs in the largest collective to hundreds of GPUs even when running a job that is tens of thousands of GPUs,” which is why their analysis focuses on…

Read the full article-specific guide →

Read the representative guide

TringT_{\mathrm{ring}}

Symbol T_ring

TrT_ring is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.

Read this term in its guide →
pp

Symbol p

p is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.

Read this term in its guide →
BB

Symbol B

B occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.

Read this term in its guide →

How to interpret it

With a fixed numerator, increasing a nonzero denominator reduces the fraction. Its accuracy depends on the assumptions and range of use described in the article.

Research cited beside this formula

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

Tring≈2 p−1p⋅NB+2 (p−1) α,T_{\mathrm{ring}} \approx 2\,\frac{p-1}{p}\cdot\frac{N}{B} + 2\,(p-1)\,\alpha,

Equation 19 · Datacenters & Infrastructure

The Physical Plant: Power, Cooling, and Networks in an AI Datacenter

This equation gives an approximation: it relates the quantities while allowing an approximation.

The cost structure of a collective is arithmetic, not policy. For a ring AllReduce over p ranks reducing N bytes at per-link bandwidth B with per-hop latency α\alpha , each rank moves Tring≈2 p−1p⋅NB+2 (p−1) αT_{\mathrm{ring}} \approx 2\,\frac{p-1}{p}\cdot\frac{N}{B} + 2\,(p-1)\,\alpha. so bandwidth cost saturates near 2N/B while the latency term grows linearly in p . Two consequences follow. The slowest link sets the pace for every rank, because the operation does not complete until all ranks have contributed. And large jobs avoid large collectives: Meta reports that multi-dimensional parallelism keeps “the number of GPUs in the largest collective to hundreds of GPUs even when running a job that is tens of thousands of GPUs,” which is why their analysis focuses on…

Meanings in this article

Equation guide → · Article →