Symbol N
N is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Read this term in its guide →Published equation contexts
Why this formula appears here
so bandwidth cost saturates near 2N/B while the latency term grows linearly in p . Two consequences follow. The slowest link sets the pace for every rank, because the operation does not complete until all ranks have contributed. And large jobs avoid large collectives: Meta reports that multi-dimensional parallelism keeps “the number of GPUs in the largest collective to hundreds of GPUs even when running a job that is tens of thousands of GPUs,” which is why their analysis focuses on collectives spanning 16 to 128 GPUs [ 6 ] .
Read the representative guide
Symbol B
B is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Read this term in its guide →How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
Research cited beside this formula
Published contexts (1)
A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.
Equation 20 · Datacenters & Infrastructure
The Physical Plant: Power, Cooling, and Networks in an AI Datacenter
This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.
so bandwidth cost saturates near 2N/B while the latency term grows linearly in p . Two consequences follow. The slowest link sets the pace for every rank, because the operation does not complete until all ranks have contributed. And large jobs avoid large collectives: Meta reports that multi-dimensional parallelism keeps “the number of GPUs in the largest collective to hundreds of GPUs even when running a job that is tens of thousands of GPUs,” which is why their analysis focuses on collectives spanning 16 to 128 GPUs [ 6 ] .
Equation guide → · Article →