≈

The passage around this formula

Analysis. The following calculation is my own, built to make the MLPerf numbers’ shape explicit rather than to reproduce them exactly. Model checkpoint writes as occurring every T hours, each costing $\tau$ hours of write time, with failures arriving as a Poisson process at fleet-wide rate $\lambda$ (failures per hour). For $\lambda$ T $\ll$ 1 , a failure inside an interval loses on average T/2 hours of recomputation, so the expected fraction of wall-clock time lost to the combination of checkpoint overhead and lost recompute is approximately $f(T) \approx \frac{\tau}{T} + \frac{\lambda T}{2}$. Minimizing over T gives an optimal interval $T^{*}$ = $\sqrt{2\tau/\lambda}$ : the checkpoint interval should shrink as the inverse square…

Read this part in the article →