Published equation contexts
Why this formula appears here
The ring algorithm is the one worth understanding mechanically, because it is bandwidth-optimal and because its cost model exposes exactly why topology matters. In a ring all-reduce across p workers, each worker is logically placed on a ring; a gradient buffer of size S is split into p chunks, and each worker simultaneously sends one chunk to its ring-neighbor while receiving a different chunk from its other neighbor, reducing (summing) as chunks arrive. A complete all-reduce takes 2(p-1) such steps, each moving S/p bytes [ 5 ] . The total data volume any one worker sends over the whole operation is:
Read the representative guide
How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
Research cited beside this formula
Published contexts (3)
A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.
Equation 4 · Datacenters
How AI Datacenter Interconnects Actually Work
This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.
The ring algorithm is the one worth understanding mechanically, because it is bandwidth-optimal and because its cost model exposes exactly why topology matters. In a ring all-reduce across p workers, each worker is logically placed on a ring; a gradient buffer of size S is split into p chunks, and each worker simultaneously sends one chunk to its ring-neighbor while receiving a different chunk from its other neighbor, reducing (summing) as chunks arrive. A complete all-reduce takes 2(p-1) such steps, each moving S/p bytes [ 5 ] . The total data volume any one worker sends over the whole operation is:
Meanings in this article
Equation guide → · Article →Equation 9 · Datacenters
How AI Datacenter Interconnects Actually Work
This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.
which is the reason ring all-reduce is called bandwidth-optimal: in the limit, total traffic per worker approaches twice the buffer size regardless of how many workers participate, rather than growing with p . What does grow with p is the number of sequential steps, 2(p-1) , and each step’s latency is bounded below by the slowest link and the slowest worker in the ring — which is precisely why NCCL switches to a tree algorithm for smaller messages and larger worker counts, trading some bandwidth efficiency for a p step count instead of a linear one [ 5 ] . This is an analytical property of the algorithm, not a vendor claim; the tree-versus-ring choice NCCL actually makes at runtime,…
Meanings in this article
Equation guide → · Article →Equation 19 · AI Hardware & Semiconductors
The Network Is the Computer Again
This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.
The bandwidth term converges to 2n as p grows — it stops depending on the number of participants — which is why the ring is the bandwidth-optimal choice for large messages. NVIDIA’s own performance documentation encodes exactly this factor, defining the bus bandwidth of an all-reduce by applying a correction of 2(p-1)/p to the naive size-over-time figure, on the reasoning that an all-reduce requires 2(p-1) data transfers across p links; all-gather, reduce-scatter and all-to-all get a factor of (p-1)/p , while broadcast and reduce get a factor of one because everything must pass through a single root [ 3 ] . The point of the correction is that bus bandwidth should stay roughly constant…
Equation guide → · Article →