addition

The passage around this formula

Pipeline parallelism splits the model by depth, giving each device a contiguous run of layers. The traffic is point-to-point rather than collective — activations forward, gradients backward, between neighbouring stages only — and the volume is modest. Its cost is not bandwidth but idleness. GPipe’s contribution was to split each batch into micro-batches so that stages could work concurrently instead of waiting for a whole batch to traverse the pipeline [ 6 ] . With p stages and m micro-batches, the standard accounting gives a bubble fraction of $\frac{p-1}{m+p-1}$. which says the only cure for pipeline idleness is more micro-batches in flight, and micro-batches cost memory. Narayanan and…

Read this part in the article →