Equation 26 · The Network Is the Computer Again
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
the writing. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
The consequence for model work is direct. Tensor parallelism produces small, frequent messages, so it lives in the latency-dominated regime where and participant count decide everything. Data parallelism produces one enormous message, so it lives in the bandwidth-dominated regime where decides everything. A fabric optimised for one is not optimised for the other, and a partitioning plan that ignores which regime each of its collectives falls into will be wrong in a way no amount of kernel tuning recovers.
Sources cited in the article section
- [2] Collective communication: theory, practice, and experience ↗
- [3] NCCL Tests — Performance Reporting and Bus Bandwidth ↗
- [1] Optimization of Collective Communication Operations in MPICH ↗
These citations give research context. Read each source to check which claims it supports.