subscript

The passage around this formula

where $w_{cp}$ is the time to write a checkpoint, $N_{\text{nodes}}$ is the job’s node count, and $r_f$ is the per-node failure rate. As $N_{\text{nodes}}$ grows, the failure rate the job experiences as a whole scales with it, which pulls the optimal interval shorter — meaning larger jobs must checkpoint more often even as each individual checkpoint write competes with the same limited storage bandwidth every other job on the cluster is also using. Reporting a real operating point, the authors note their clusters run at an Effective Training Time Ratio, their measure of time actually spent making forward progress, of 0.85 to 0.9 for their largest and highest-priority jobs even with hourly checkpoint…

Read this part in the article →