=

The passage around this formula

The classical accounting for a collective operation separates three terms: a per-message latency, a per-byte transfer cost, and a per-byte computation cost for reductions [ 2 ] . Writing $\alpha$ for latency per message, $\beta$ for transfer time per byte and $\gamma$ for reduction time per byte, the cost of a collective over p participants on n bytes takes the shape $T(n,p) = \alpha\,L(p) + \beta\,W(n,p) + \gamma\,C(n,p)$. where L counts message steps, W counts bytes crossing each link and C counts arithmetic. The whole art of collective algorithm design is choosing which of these terms to pay.

Read this part in the article →