AI Datacenter Systems Engineering in 2035: Scenarios, Signals, and Falsifiable Predictions

Checkpoint intervals are already timed in minutes, and the field has not agreed whether the fix is a smarter scheduler or a paradigm that stops needing checkpoints. Five trajectories to 2035, each with what would prove it wrong.

A closed-loop scheduler that detects, evicts, and reroutes without a human in the loop is still a bench prototype; the relay armature here stands in for the decision. — Image prompt and art direction by Brecht Corbeel; image generated to that direction.

Abstract

This article builds dated, falsifiable scenarios for where AI cluster systems engineering goes by 2035, organised around five trajectories: how far reactive cluster management extends toward closed-loop self-healing scheduling; whether checkpoint-free fault tolerance built on redundant computation displaces periodic checkpointing or merely supplements it; how far fleet-scale telemetry can move failure handling from post-hoc diagnosis to precursor-based prediction; how checkpoint and storage architecture must change as training runs and clusters grow; and what would force a discontinuity in the underlying fault-tolerance paradigm. Each trajectory carries a horizon, assumptions, indicators, and a disconfirmation condition, grounded in reliability studies from Meta and Shanghai AI Laboratory, checkpointing systems from ByteDance, Microsoft Research, and academic groups, and MLCommons' first storage-checkpointing benchmark. This article holds power delivery and cooling fixed and out of scope, since those belong to this series' pilot article rather than this one.

Five trajectories, and what would prove each one wrong

This series’ pilot article treated an AI datacenter as a physical plant: a substation, a heat-rejection system, and a fabric, with the accelerators as the load hung in the middle of it. That framing was deliberate, and this article holds it fixed. Power delivery and cooling are engineering problems with their own literature, and they are not this piece’s subject. This piece’s subject is what happens once the power is flowing and the heat is leaving the building: how a cluster of tens or hundreds of thousands of accelerators is scheduled, kept running, checkpointed, watched, and eventually rebuilt when a piece of it inevitably breaks.

That subject has its own five-trajectory shape. This article asks, for five specific and load-bearing pieces of cluster systems engineering, what happens between now and 2035, and commits in advance to the observations that would show each answer wrong. The five are: how far reactive cluster management extends toward a genuinely closed-loop, self-healing scheduler; whether checkpoint-free fault tolerance built on redundant computation displaces periodic checkpointing or merely supplements it in specific regimes; how far fleet-scale telemetry can move failure handling from post-hoc diagnosis toward precursor-based prediction; how checkpoint and storage architecture must change as model size and cluster size keep growing; and what would force a genuine discontinuity in the underlying fault-tolerance paradigm — a change sharp enough that today’s roadmaps stop being useful guides.

Throughout, five kinds of statement are kept visibly separate. A fact is something disclosed in a peer-reviewed paper, a production engineering report, or a published benchmark methodology. A vendor claim is a company’s own statement about its own system, reported as a claim because the company has an interest in the answer. Analysis works out a consequence of stated facts, including original calculations clearly marked as such. A scenario is one internally consistent way the future could go, presented alongside its alternatives rather than as the likely one. A prediction commits to a horizon, states its assumptions plainly, names an observable indicator, and states in advance what would disconfirm it. None of the five trajectories below is a straight-line forecast; each is a documented present extended under a stated assumption, with the assumption itself treated as the interesting and doubtable part.

The documented present

As of this writing, cluster-scale reliability engineering has moved from an afterthought to a discipline with its own production literature. Meta’s analysis of two large-scale ML research clusters covers eleven months of operation, more than 150 million A100 GPU-hours, and roughly 4 million jobs; it introduces a taxonomy of failure types, fits a model projecting Mean Time to Failure across GPU scales, and defines an Effective Training Time Ratio measuring how much of a job’s wall-clock time actually advances training rather than being consumed by failure and recovery [3]. Shanghai AI Laboratory’s characterization of six months of large-language-model development traffic across its Seren and Kalos clusters, 4,704 A100 GPUs combined, documents how LLM workloads’ resource-utilization and failure patterns diverge from earlier deep-learning workloads, and reports building a fault-tolerant pretraining pipeline with automatic failure diagnosis in response [6]. Underneath both studies sits a hardware-level fact predating the LLM era: silent data corruption, in which a defective component produces an incorrect result without tripping any built-in error check, was already being tracked across hundreds of thousands of machines and monitored for over eighteen months in Meta’s production fleet, with the company concluding that no single layer — hardware resiliency, production-level detection, or fault-tolerant software alone — is sufficient [12].

Against that backdrop, the systems response has been an explosion of purpose-built infrastructure. ByteDance’s MegaScale reports training a 175-billion-parameter model across 12,288 GPUs at 55.2% model FLOPs utilization, a 1.34-times improvement over Megatron-LM, and states that “many hard stability issues only emerge at large scale,” which is why the system ships its own diagnostic tooling for identifying root causes and mitigating stragglers [1]. Recovery-specific systems have proliferated alongside it: Gemini checkpoints to host CPU memory every iteration and reports recovering from failure more than 13 times faster than disk-based approaches [2]; Microsoft Research’s Just-In-Time Checkpointing writes a checkpoint only when a failure is detected rather than on a fixed schedule, claiming recovery cost falls “from several minutes to a few seconds per GPU, with nearly zero steady state overhead” [4]; and Alibaba’s Unicron reports a 1.9-times training-efficiency improvement on a 128-GPU cluster through in-band error detection and cost-aware reconfiguration [8]. None of this is speculative. It is the documented baseline the five trajectories below extend.

Trajectory one: from reactive scripts to closed-loop self-healing scheduling

Fact. Production fault-tolerance systems already close part of the loop. Unicron’s error detection runs “in-band,” meaning it observes training itself for signs of trouble rather than waiting for an external monitor, and its reconfiguration planner is described as “dynamic” and “cost-aware,” choosing among available recovery actions rather than following one fixed script [8]. Shanghai AI Laboratory’s Acme clusters run an automatic recovery pipeline that performs “LLM-involved failure diagnosis” before restarting a job [6]. Meta’s Effective Training Time Ratio gives operators a single number to optimize against — the fraction of wall-clock time a job actually spends making training progress — which is exactly the objective a closed-loop scheduler would be built to maximize [3].

Figure 1. Self-healing at fleet scale starts before a node ever runs a job — a newly built machine is proven on a bench like this one, or it is not admitted. — Image prompt and art direction by Brecht Corbeel; image generated to that direction.

Analysis. “Self-healing” is doing two different jobs in these systems today, and conflating them overstates how autonomous any of them currently is. The first job is detection and reaction: noticing a straggler, a hang, or an error signature, and triggering a known remediation — restart this rank, exclude this node, reroute this pipeline stage. All three systems above do this. The second job is genuinely closed-loop planning: choosing among reconfigurations by predicted cost without a human pre-enumerating the decision tree, and — the harder step — deciding whether a newly built or repaired node should be trusted with production work at all. That second capability exists in narrower form (Unicron’s plan generation; commissioning pipelines that gate new hardware behind diagnostic burn-in) but nothing cited here describes a scheduler unifying failure response, capacity planning, and hardware admission into one continuously learning fleet-wide policy.

Prediction. Horizon: end of 2031. Production AI clusters at major operators will run a scheduler that autonomously chooses among multiple remediation strategies (restart-in-place, node exclusion, pipeline reconfiguration, job migration) based on a learned or continuously updated cost model, rather than a fixed priority order of remediations, for the majority of large training jobs. Assumption: the economic pressure visible in Unicron’s and MegaScale’s own framing — that failure-related cost, not just failure-related downtime, is now large enough to justify dedicated systems engineering — continues to grow roughly with cluster size rather than plateauing once today’s point solutions are deployed. Indicator: published operator engineering reports and systems papers describing multi-strategy, cost-aware remediation selection as a default behavior rather than a research contribution. Disconfirmed if, by the horizon, production schedulers at major operators still select among remediations by a fixed, human-authored priority order for the majority of large jobs, with cost-aware planning remaining confined to published research prototypes.

Trajectory two: checkpoint-free fault tolerance, and the disagreement underneath it

Fact. Two different technical bets on removing checkpointing from the critical recovery path already exist in the peer-reviewed literature, and they are not the same bet. Bamboo inserts redundant computation into the natural idle gaps of pipeline-parallel training — a node computes not only its own pipeline stage but also a neighbor’s, filling “pipeline bubbles” that would otherwise be wasted — and reports 3.7 times higher throughput than checkpointing-based recovery and 2.4 times lower cost than using only on-demand instances, in a setting built around cheap, frequently preempted cloud instances [11]. Oobleck instead pre-computes a set of heterogeneous pipeline-parallel templates and instantiates enough logically redundant pipeline replicas that any f-or-fewer simultaneous node failures still leave a valid combination of templates covering all remaining resources, reporting up to 29.6 times higher throughput than Bamboo and the earlier Varuna system in its own evaluation [7]. Neither system eliminates checkpointing as a durability mechanism for the trained model itself; both aim to eliminate it as the primary recovery path for an individual node failure.

Figure 2. Redundant computation trades a checkpoint you might not reach in time for a second machine doing some of the same work all along; this bench rig is where that trade is still being tested rather than assumed. — Image prompt and art direction by Brecht Corbeel; image generated to that direction.

Analysis. This is a genuine, unresolved disagreement about where resilience should live, not a solved problem with two implementations. One position, visible in Gemini and Just-In-Time Checkpointing, keeps periodic or event-triggered checkpointing as the primary recovery mechanism and attacks the cost of using it — write checkpoints somewhere faster than durable storage, or write them only exactly when needed [2, 4]. The other, visible in Bamboo and Oobleck, treats checkpointing itself as the avoidable step and instead spends idle compute or extra replicas continuously so a failure never forces a rollback at all. The two camps look for progress in different places: the first watches recovery time and checkpoint overhead fall toward zero; the second watches redundant-compute overhead fall while the replica count needed for a given fault tolerance shrinks. This article does not adjudicate which is correct, because the evidence does not yet let it — Bamboo’s and Oobleck’s gains are demonstrated on preemptible cloud instances and pipeline-parallel topologies specifically, not yet shown to generalize to every parallelism strategy at every scale.

Prediction. Horizon: end of 2032. At least one major operator will run production training at greater than 50,000-accelerator scale in which redundant computation or replica-based recovery, rather than checkpoint-and-restart, is the primary mechanism for tolerating a single-node failure without measurable throughput loss. Assumption: the idle-bubble and replica-redundancy techniques demonstrated at hundreds-to-low-thousands of GPUs in the cited papers continue to find enough spare compute or acceptable redundancy overhead as parallelism strategies and cluster sizes grow, rather than the redundancy cost itself growing faster than cluster size. Indicator: operator engineering reports or systems papers describing single-node failures absorbed without a checkpoint-restore event at production LLM-training scale. Disconfirmed if, by the horizon, published large-scale training reports at that scale still describe checkpoint-and-restart, however fast, as the primary recovery mechanism for ordinary single-node failures.

Trajectory three: from post-hoc diagnosis to precursor-based prediction

Fact. Failure at fleet scale is not a rare event to be handled exceptionally; it is a continuous background process to be engineered around. MLCommons’ first industry-standard checkpointing benchmark states plainly that “at the scale of computation being implemented for training large AI models, regular component failures are simply a fact of life,” and its modeling projects roughly 7.67 failures per day at 16,000 accelerators — close enough to Meta’s separately reported 7.76 failures per day during Llama 3 training that MLCommons cites it as a cross-check — rising sharply at larger scale, with a 100,000-accelerator cluster seeing a failure roughly every 30 minutes and a hypothetical million-accelerator system roughly one every three minutes [9]. Silent data corruption compounds the diagnostic problem because it produces no error signal at all — Meta’s fleet-scale study describes defective components “bypassing” existing CPU error-detection mechanisms entirely, so the failure exists before any monitoring system registers it [12].

Figure 3. Predictive detection depends on a precursor signal existing before the failure does; this bench is where that signal is still being hunted for, one sensor at a time. — Image prompt and art direction by Brecht Corbeel; image generated to that direction.

Analysis. The following is my own structural analysis, not drawn from any single cited source. If a fleet of N identical devices each fails independently at rate \lambda, the fleet-wide failure rate is N\lambda, so

\mathrm{MTTF}_{\mathrm{fleet}} \approx \frac{1}{N\lambda} = \frac{\mathrm{MTTF}_{\mathrm{device}}}{N}.

That relation is what licenses treating fleet-scale failure as a continuous background rate rather than a sequence of independent surprises, and it is the same move Meta’s reliability study makes in fitting a model to project Mean Time to Failure across GPU scales [3]. But it depends entirely on independence, and the sources above describe two ways that assumption breaks. First, correlated failure: a batch sharing a manufacturing defect or an upstream dependency fails together, precisely the shared-cause pattern Meta’s silent-data-corruption study traces to specific defective production lots rather than random wear-out [12]. Second, and more interesting for prediction specifically, \lambda is not actually constant — real components have a rising hazard rate as they approach failure, which is the entire premise of precursor-based detection: if degradation produces an observable signal (vibration, current draw, correctable-error rate, thermal drift) before the hard failure, \lambda for that unit is knowably higher than the fleet average beforehand, and a system reading that signal can act before rather than after. Nothing cited here demonstrates that capability in production; the cited systems demonstrate fast diagnosis and recovery after a failure, a different and easier-to-verify claim.

Prediction. Horizon: end of 2033. Precursor-based failure prediction — flagging a specific unit as elevated-risk from telemetry before it hard-fails, with enough lead time to drain or replace it pre-emptively — will be in routine production use at major operators for at least one major failure category (for example correlated-error precursors to a hard fault), but will not have eliminated the need for fast post-hoc recovery, because not all failure modes produce a usable precursor signal. Assumption: at least some economically important failure modes have a detectable degradation signature with sufficient lead time, an assumption the cited sources do not settle either way. Indicator: operator engineering reports disclosing pre-emptive replacement rates driven by telemetry-based risk scoring, separate from and in addition to post-failure recovery statistics. Disconfirmed if, by the horizon, published fleet reliability reports still describe failure handling exclusively in post-hoc terms — detection after the fact and recovery afterward — with no disclosed pre-emptive, precursor-driven replacement program at any major operator.

Trajectory four: storage and checkpoint architecture at ever-larger scale

Fact. Checkpoint size and required throughput are both growing directly with model and cluster size, and the growth is documented rather than merely intuitive. MLCommons’ benchmark specifies checkpoint sizes of 105 GB for an 8-billion-parameter model, 912 GB for 70 billion parameters, 5.29 TB for 405 billion parameters, and 15 TB for a trillion-parameter model, and reports that holding progress loss to 5% or less requires checkpoint intervals to tighten from roughly every 9.3 minutes at 16,000 accelerators to roughly every 1.5 minutes at 100,000 — at which point a trillion-parameter model generates more than 14 petabytes of checkpoint writes per day, requiring on the order of 3.6 terabytes per second of sustained throughput across the cluster [9]. The systems response has been architectural rather than purely capacity-driven. Check-N-Run, aimed at recommendation models whose embedding tables are what actually changes each step, uses differential checkpointing of only the modified portion plus quantization to cut required write bandwidth 6 to 17 times and required capacity 2.5 to 8 times [5]. ByteCheckpoint generalizes further with a parallelism-agnostic checkpoint representation that lets a checkpoint be reshaped across different parallelism configurations at load time, reporting a 54.2-times reduction in runtime checkpoint stalls and up to 9.96 and 8.80 times faster saving and loading respectively against open-source baselines [10].

Figure 4. A checkpoint is only as fast as the medium it lands on; what that medium should be at ever-larger scale is still being characterized here, one sample at a time. — Image prompt and art direction by Brecht Corbeel; image generated to that direction.

Analysis. The following calculation is my own, built to make the MLPerf numbers’ shape explicit rather than to reproduce them exactly. Model checkpoint writes as occurring every T hours, each costing \tau hours of write time, with failures arriving as a Poisson process at fleet-wide rate \lambda (failures per hour). For \lambda T \ll 1, a failure inside an interval loses on average T/2 hours of recomputation, so the expected fraction of wall-clock time lost to the combination of checkpoint overhead and lost recompute is approximately

f(T) \approx \frac{\tau}{T} + \frac{\lambda T}{2}.

Minimizing over T gives an optimal interval T^{*} = \sqrt{2\tau/\lambda}: the checkpoint interval should shrink as the inverse square root of the fleet failure rate, not linearly with it, so frequency rises roughly as \sqrt{N} purely from the failure-rate side as accelerator count N grows. Comparing that to MLPerf’s own reported tightening — interval falling from 9.3 to 1.5 minutes, a 6.2-times increase in frequency, as scale rises 6.25-times from 16,000 to 100,000 accelerators — the real tightening is close to linear in scale, noticeably steeper than the \sqrt{N} this simple model predicts. That gap is informative rather than a contradiction: it is consistent with operators holding a fixed acceptable progress-loss percentage (MLPerf specifies 5%) rather than freely re-optimizing T against \tau and \lambda jointly, and with \tau itself growing as checkpoints get larger at larger scale — both push the real interval down faster than a pure failure-rate argument would. None of the cited sources explains that gap directly; this derivation is offered to sharpen the open question, not to answer it.

Prediction. Horizon: end of 2030. Checkpoint storage for frontier-scale training will have shifted from general-purpose distributed filesystems toward storage architectures purpose-built for the checkpoint write-and-reshard pattern — differential, quantized, or parallelism-agnostic by design, following Check-N-Run’s and ByteCheckpoint’s direction — as the default rather than the exception at major operators running trillion-parameter-class training. Assumption: checkpoint size continues to scale with parameter count roughly as documented, without a change in optimizer or precision strategy that structurally shrinks checkpoint size faster than model size grows. Indicator: published or benchmarked storage-system architectures for frontier training runs, and MLPerf Storage’s own future checkpointing-round results across submitting organizations. Disconfirmed if, by the horizon, the majority of disclosed frontier-scale training checkpoint systems are still built on unmodified general-purpose distributed filesystems without differential, quantized, or resharding-aware checkpoint representations.

Trajectory five: what could force a discontinuity

Scenario. All four trajectories above are continuations: faster recovery, smarter scheduling, earlier detection, better-architected storage. A genuine discontinuity would be a change to the underlying fault-tolerance paradigm itself, and the clearest candidate visible in the current literature is the disagreement already named in trajectory two, pushed to its logical extreme: a training method that does not require tight, frequent synchronization across the whole fleet at all, and therefore has no single global state that a failure can corrupt in the first place. Bamboo’s and Oobleck’s redundant-computation and replica-template approaches are early, partial steps in that direction — they reduce how much a single failure costs, but training remains synchronous and globally coordinated at the level the parallelism strategy demands [11, 7]. A full discontinuity would go further: a training regime where nodes tolerate being loosely, rather than tightly, synchronized as a matter of algorithm design, so that losing one participant briefly degrades progress rather than halting or corrupting the whole job. Nothing in the sources behind this article demonstrates that regime operating at frontier scale in production; it is named here specifically because a genuine discontinuity, by definition, would not yet be visible as a mainstream production practice.

Figure 5. Every trajectory in this article is a claim pinned up before the date it names has arrived; a forced discontinuity would tear the sheet rather than add a card to it. — Image prompt and art direction by Brecht Corbeel; image generated to that direction.

Analysis. The pressure toward such a change is at least partly structural, not merely a matter of taste. MegaScale’s own account of operating at 12,288 GPUs states plainly that “many hard stability issues only emerge at large scale” [1], and Meta’s reliability study finds large jobs disproportionately exposed to failure despite being a minority of jobs by count [3] — both documented facts, and both pointing toward tight global synchronization growing proportionally more fragile as fleets grow, independent of any single vendor’s roadmap. Whether that pressure resolves through better engineering of the synchronous model (trajectories one through four) or through abandoning tight synchronization as the default is exactly the disagreement this article declines to adjudicate: the evidence does not yet distinguish “the synchronous model is being engineered hard enough that it holds” from “it is accumulating enough friction that it eventually gives way.”

Prediction. Horizon: end of 2035. No major operator will have shipped a frontier-scale (greater than 100,000-accelerator) production training run in which loose, algorithm-level fault tolerance — rather than fast synchronous recovery via checkpointing, redundant computation, or replica templates — is the primary mechanism handling routine node failure. This is stated as the conservative, continuation-favoring expectation specifically so the discontinuity scenario has a clean test. Assumption: the systems-engineering responses in trajectories one through four continue to absorb rising failure rates faster than loosely-synchronized training methods close the efficiency gap that currently makes tight synchronization worth defending. Indicator: operator disclosures of the fault-tolerance mechanism actually used as the primary line of defense in their largest production training runs. Disconfirmed if, before the horizon, a major operator discloses a frontier-scale production run in which loose or algorithm-level fault tolerance, rather than a synchronous recovery mechanism, is the documented primary defense against routine node failure.

Two axes, four scenarios toward 2035

Crossing trajectory five’s synchronous-versus-loose question with trajectories one through four’s smooth-continuation-or-discontinuity question gives four scenarios. None is presented as the likely one.

Scenario A: Engineered Synchrony. Tight, globally synchronized training remains the default through 2035; the pressure identified above is absorbed by continued improvement along trajectories one through four alone. Horizon: through 2035. Assumptions: the efficiency gap between tightly and loosely synchronized training stays large enough that no operator finds the trade worthwhile at frontier scale. Indicators: continued incremental improvement in recovery time, Effective Training Time Ratio, precursor lead time, and checkpoint throughput, with no operator disclosing a shift away from synchronous training. Disconfirmation: falsified if any major operator discloses loose, algorithm-level fault tolerance as the primary mechanism at frontier scale before 2035.

Scenario B: Partial Redundancy. Checkpoint-free, redundancy-based recovery becomes the production default for absorbing single-node failures, but the underlying training stays tightly synchronized in every other respect — redundancy changes how a failure is survived, not how the job is structured. Horizon: through 2032, per trajectory two. Assumptions: redundant-compute overhead scales acceptably with cluster size. Indicators: production disclosures matching trajectory two’s indicator, with no accompanying synchronization-strategy change. Disconfirmation: falsified if redundancy-based recovery misses production-default status by its horizon, or arrives bundled with a synchronization change (indicating scenario D instead).

Scenario C: Detected, Not Prevented. Precursor-based prediction matures and becomes routine, changing when a failure is handled — pre-emptively rather than reactively — without changing how the resulting gap is absorbed, which still relies on the synchronous mechanisms of trajectories one, two, and four. Horizon: through 2033, per trajectory three. Assumptions: enough failure modes carry a usable precursor signal to matter at fleet scale, without motivating a structural training change. Indicators: disclosed pre-emptive replacement programs alongside unchanged synchronous recovery architecture. Disconfirmation: falsified if precursor prediction misses routine production use by its horizon, or its arrival coincides with a synchronization change.

Scenario D: Forced Divergence. A genuine discontinuity occurs before 2035 — loose, algorithm-level fault tolerance becomes the primary mechanism at frontier scale, the structural pressure named in trajectory five outrunning the engineering responses in trajectories one through four. Horizon: a disclosed instance before end of 2035. Assumptions: the efficiency gap between synchronous and loosely-synchronized training closes faster than continuation-favoring engineering can defend tight synchronization at the largest scales. Indicators: a named operator disclosure of loose fault tolerance as the primary defense above 100,000-accelerator scale, not a research prototype or sub-frontier deployment. Disconfirmation: falsified if no such disclosure occurs by the horizon — the conservative expectation under trajectory five’s own prediction.

What survives across all four cells, and the wildcard none of them names

One thing holds regardless of which cell the industry lands in: the independence assumption behind \mathrm{MTTF}_{\mathrm{fleet}} \approx \mathrm{MTTF}_{\mathrm{device}}/N keeps breaking as fleets grow, whether the response is better synchronous engineering or a move away from synchrony altogether. Correlated failure — a bad manufacturing lot, a shared dependency, a batch of units commissioned and aging together — is documented independently by Meta’s silent-data-corruption study [12] and implicit in the taxonomy of failure types in Meta’s reliability study [3]. That pressure exists in every scenario above, which is why fault-domain design — structuring which units share which dependencies, so correlated failure stays contained rather than fleet-wide — sits underneath all five trajectories rather than being a sixth one.

The wildcard none of the four scenarios names is a fault-tolerance model resembling neither camp in trajectory two’s disagreement: not faster checkpointing, not redundant computation, not looser synchronization, but something that removes the concept of a “failure event” from the systems picture entirely — hardware degrading gracefully enough, and instrumented finely enough, that trajectory three’s precursor detection becomes so reliable that nothing an operator would currently call a “failure” happens without advance notice and an already-arranged replacement. Nothing in the sources behind this article points to that capability existing even at prototype stage; it is named here only because a genuine discontinuity, by definition, is not visible in advance from inside the roadmaps it would disrupt.

What to take away

Five things are true about AI cluster systems engineering at the point this article was written, and none is a prediction: fleet-scale failure is already a continuous background process, documented at rates from single digits to hundreds per day depending on scale; production systems already close part of the self-healing loop, but no scheduler unifying detection, remediation planning, and hardware admission into one continuously learning fleet-wide policy has been documented; two unreconciled technical bets exist for removing checkpoint-and-restart from the critical recovery path, and the evidence does not yet favor one; precursor-based failure prediction is conceptually sound but not yet documented in routine production use; and checkpoint storage architecture is already being rebuilt around the checkpoint-and-reshard problem rather than treated as generic file storage. Everything beyond those five facts here is either a vendor’s own claim, an analysis worked out from stated facts (including two calculations performed for this article and marked as such), or a scenario and prediction carrying an explicit horizon and an explicit way to be proven wrong. The discipline that matters most for a reader returning to this piece nearer 2035 is not which scenario turned out closest to right — it is checking each prediction’s stated indicator against what actually happened, and noticing which assumptions held and which quietly stopped being true.

Sources

1. Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, and Xin Liu. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs. USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2024). DOI: 10.48550/arXiv.2402.15627.
2. Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, T. S. Eugene Ng, and Yida Wang. Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints. ACM Symposium on Operating Systems Principles (SOSP) (2023). DOI: 10.1145/3600006.3613145.
3. Apostolos Kokolis, Michael Kuchnik, John Hoffman, Adithya Kumar, Parth Malani, Faye Ma, Zachary DeVito, Shubho Sengupta, Kalyan Saladi, and Carole-Jean Wu. Revisiting Reliability in Large-Scale Machine Learning Research Clusters. arXiv (2024). DOI: 10.48550/arXiv.2410.21680.
4. Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav S. Gulavani, Nipun Kwatra, Ramachandran Ramjee, and Muthian Sivathanu. Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures. ACM European Conference on Computer Systems (EuroSys) (2024). DOI: 10.1145/3627703.3650085.
5. Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, and Murali Annavaram. Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models. USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2022). DOI: 10.48550/arXiv.2010.08679.
6. Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, Yonggang Wen, and Tianwei Zhang. Characterization of Large Language Model Development in the Datacenter. USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2024). DOI: 10.48550/arXiv.2403.07648.
7. Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury. Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates. ACM Symposium on Operating Systems Principles (SOSP) (2023). DOI: 10.1145/3600006.3613152.
8. Tao He, Xue Li, Zhibin Wang, Kun Qian, Jingbo Xu, Wenyuan Yu, and Jingren Zhou. Unicron: Economizing Self-Healing LLM Training at Scale. arXiv (2024). DOI: 10.48550/arXiv.2401.00134.
9. MLCommons. Announcing the MLPerf Storage v2.0 Checkpointing Workload. MLCommons (2025).
10. Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, Xin Liu, and Chuan Wu. ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development. USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2025). DOI: 10.48550/arXiv.2407.20143.
11. John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2023). DOI: 10.48550/arXiv.2204.12013.
12. Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, and Sriram Sankar. Silent Data Corruptions at Scale. arXiv (2021). DOI: 10.48550/arXiv.2102.11245.

Originally published at https://absolutedigitalpublishers.com/articles/ai-datacenter-systems-engineering-in-2035-scenarios-signals-and-falsifiable-predictions.