Equation 3 · Part 7 · The Network Is the Computer Again
Denominator: m+p-1
What this part means
The complete quantity below the fraction bar; it must be nonzero for this division.
Its job in the formula
m+p-1 occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.
Full expression→Denominator: m+p-1→Article meaning
The passage around this formula
Pipeline parallelism splits the model by depth, giving each device a contiguous run of layers. The traffic is point-to-point rather than collective — activations forward, gradients backward, between neighbouring stages only — and the volume is modest. Its cost is not bandwidth but idleness. GPipe’s contribution was to split each batch into micro-batches so that stages could work concurrently instead of waiting for a whole batch to traverse the pipeline [ 6 ] . With p stages and m micro-batches, the standard accounting gives a bubble fraction of . which says the only cure for pipeline idleness is more micro-batches in flight, and micro-batches cost memory. Narayanan and…
Learn the underlying idea
A fraction a/b means a divided by b. The top number is the numerator; the bottom number is the denominator, and it cannot be zero.
Open the illustrated fractions: division written vertically guide →
Sources cited in the surrounding passage
- [6] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism ↗
- [5] Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM ↗
These citations provide research context; check each source for the exact claim it supports.