What does this equation mean?

Pipeline parallelism splits the model by depth, giving each device a contiguous run of layers. The traffic is point-to-point rather than collective — activations forward, gradients backward, between neighbouring stages only — and the volume is modest. Its cost is not bandwidth but idleness. GPipe’s contribution was to split each batch into micro-batches so that stages could work concurrently instead of waiting for a whole batch to traverse the pipeline [ 6 ] . With p stages and m micro-batches, the standard accounting gives a bubble fraction of $\frac{p-1}{m+p-1}$. which says the only cure for pipeline idleness is more micro-batches in flight, and micro-batches cost memory. Narayanan and colleagues composed tensor, pipeline and data parallelism together, reported 502 petaFLOP/s on 3,072 GPUs at 52 percent of theoretical peak, and introduced an interleaved pipeline schedule they report improves throughput by over ten percent at comparable memory [ 5 ] .