AI Datacenter Systems Engineering in Practice: An Advanced Technical Guide

Operating a cluster is a different job from building one

A cluster procurement document ends at commissioning: the racks are powered, the fabric passes its acceptance tests, the storage array reports its rated throughput, and the site is handed to whoever runs it. Everything after that handover is a separate discipline with its own decisions, made repeatedly rather than once — how often to checkpoint, what counts as a fleet-health signal worth waking someone up for, how large a blast radius any single fault is allowed to have, how one scheduler serves both a synchronous training job and a latency-bound inference service from the same accelerators, and how much storage headroom to hold in reserve for the day the checkpoint writes and the data pipeline collide.

This guide is written from the operator’s side of that handover. It draws on published engineering accounts of clusters that were actually run at scale — Meta’s Llama 3 training infrastructure, ByteDance’s MegaScale system, Microsoft’s Philly cluster traces, Alibaba’s and Google’s cluster-scheduler research, SenseTime’s Acme datacenter trace, Meta’s own multi-cluster reliability study, and Imbue’s public account of standing up a 4,088-GPU cluster from bare metal — rather than on vendor marketing about what a cluster can theoretically do. The claims below are attributed to the specific study or operator that made them; where a number is a projection rather than an observation, that distinction is kept explicit, because the difference matters when it is your fleet.

Checkpoint frequency as a computed policy, not a habit

The naive approach to checkpointing is a fixed interval chosen once and left alone — every fifteen minutes, every hour, whatever felt safe during the pilot run. That approach breaks as clusters grow, because the failure rate a checkpoint policy has to survive is not a property of the model or the team; it is a property of the accelerator count, and it moves fast.

MLCommons made this concrete when it added a checkpointing workload to the MLPerf Storage v2.0 benchmark suite in 2025, framing the problem directly: “due to the cluster failure rate scaling to the power of cluster size, MTTF exponentially decreases as cluster size increases” ^[8]. Working from Meta’s own published Llama 3 training data, MLCommons’s modelling implies that at a scale of 100,000 accelerators, holding checkpoint overhead below 5% of training time requires roughly 967 checkpoints per day — a checkpoint every 1.5 minutes — with each individual save completing in about 4.4 seconds ^[8]. That is not a design choice a team can defer; it is arithmetic that a storage system either supports or does not.

The underlying trade-off is a classical one from checkpoint/restart theory, and it is worth making explicit because it is the model every specific policy below is an instance of. If a checkpoint costs a fixed time $C$ to write and failures arrive with mean time between them $M$, then choosing a checkpoint interval $T$ trades two costs against each other: writing more often burns time on overhead, and writing less often burns time re-doing work lost since the last save. Minimising the sum of those two costs — checkpoint overhead $C/T$ plus expected rework $T/2M$ — over $T$ gives the classical optimum:

The consequence that matters operationally is that $T_{\text{opt}}$ shrinks as $M$ shrinks, and $M$ shrinks sharply as accelerator count grows — so a checkpoint interval tuned for a 512-GPU job is not a conservative choice for a 16,000-GPU job, it is simply wrong, and it will keep being wrong in the same direction as the fleet grows further. The only way to hold $T_{\text{opt}}$ steady as $M$ falls is to drive $C$ down, which is why so much of the published checkpointing literature is really about reducing checkpoint cost rather than changing checkpoint frequency directly.

Meta’s own account of training the 405B-parameter Llama 3 model on up to 16,000 H100 GPUs describes exactly this response: the checkpointing system was engineered to minimise “the GPU pause time” while “increasing checkpoint frequency to reduce the amount of lost work after a recovery” ^[2]. Check-N-Run, Meta’s earlier checkpointing system built for large recommendation models, attacks the same $C$ term directly through two techniques — differential checkpointing that tracks and saves only the portion of the model that changed, and quantization of the saved values — reporting a 6-to-17x reduction in required write bandwidth and a 2.5-to-8x reduction in required storage capacity on production models, without degrading trained accuracy ^[5]. SenseTime’s Acme cluster took a related but distinct approach for their fault-tolerant pretraining system, using an asynchronous checkpointing strategy that reported a 3.6-to-58.7x speedup in checkpoint overhead across 7B and 123B-parameter models ^[4]. These are three different engineering answers to the same $C$ term, reported by three different operators on their own workloads — not a ranking, since none of the three published comparable numbers on the same hardware and model.

The reliability payoff of getting this right is measurable in terms operators actually track. Meta’s own multi-cluster reliability study defines Effective Training Time Ratio (ETTR) — roughly, the fraction of wall-clock time a job spends making forward progress rather than recovering — and reports observed ETTR of approximately 0.85 to 0.90 for their largest production jobs in the 2,048-to-4,096-GPU range ^[3]. Using their fitted model, the same study estimates that a hypothetical 12,000-GPU run would need checkpoint write overhead reduced to roughly 10 seconds, or the cluster’s underlying failure rate improved from about 6.50 to about 1 failure per thousand node-days, to hold ETTR at 0.9 — two different levers arriving at the same target, offered as the paper’s own modelled scenario rather than a measurement of an existing fleet ^[3]. The policy conclusion is that checkpoint frequency is not set once at project kickoff; it is recomputed whenever accelerator count, model size, or measured failure rate changes, using the operator’s own current MTTF rather than an inherited assumption.

Fleet health telemetry: catching the failure that does not crash anything

A checkpoint policy assumes failures announce themselves — a job crashes, a node drops off the network, recovery kicks in. The harder operational problem is the failure that does not announce itself at all: a GPU that runs correctly but slowly, a memory module returning occasionally wrong values, a network link that degrades without dropping. The systems-reliability literature calls this class “fail-slow,” and it predates the AI cluster era. Gunawi and colleagues’ study of 114 fail-slow incidents across 14 large production deployments found that every major hardware category — disk, SSD, CPU, memory, and network — can exhibit performance faults rather than clean stop failures, that faults convert from one form into another as they propagate, and that root causes can cascade for a long chain before producing a visible symptom ^[10]. Their examples are the reason fleet telemetry cannot simply watch for red lights: a faulty motherboard sensor reporting a false temperature reading caused a set of CPUs to silently downclock into energy-saving mode, and in another deployment a single memory card running at 25% of its rated speed caused a cascading backlog of out-of-memory errors and crashes in the service built on top of it ^[10]. Neither failure would trip a simple up/down health check; both were slow enough to look like ordinary variance until someone went looking.

Production LLM training systems have built specific tooling in response. ByteDance’s MegaScale paper reports building “a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers” as a core part of scaling training past 10,000 GPUs, treating this diagnostic layer as inseparable from the training system itself rather than as an add-on ^[1]. SenseTime’s Acme system pushes the diagnosis step further by having an LLM assist the triage process — a two-step pipeline of rule-based filtering followed by vector-similarity retrieval against known failure signatures — which the authors report reduced the manual-intervention burden on their engineering team by roughly 90% ^[4]. Imbue’s public account of operating a 4,088-GPU, 511-node H100 cluster describes a comparable but more conventional telemetry stack built from Prometheus and Grafana for metrics, Loki for log aggregation, a DCGM exporter for GPU-specific signals such as clock throttling events, and a node exporter for general hardware and OS state, layered with custom tooling that captures stack traces during unexpectedly slow batches and parses switch event logs for early signs of fabric trouble ^[7].

What separates telemetry that catches fail-slow behaviour from telemetry that only catches outright crashes is the presence of a performance baseline per component, not just a liveness check. Imbue’s health-check regime runs GPU diagnostics that verify expected GPU count, monitor ECC error counts, and validate NVLink topology; checks storage utilisation against a threshold; and validates InfiniBand link error rates and firmware versions — but the team also runs extended multi-node diagnostic passes lasting 12 to 24 hours specifically because intermittent, fail-slow-type issues do not always show up in a quick pass ^[7]. Alerting policy follows from the same logic: a threshold set to catch only total failure will miss exactly the fail-slow class that the literature says dominates hard-to-diagnose incidents, while a threshold set too sensitively drowns operators in false positives from ordinary variance. Neither MegaScale, Acme, nor Imbue’s account publishes a single universal threshold, because none of them found one — each tuned thresholds against its own hardware generation and workload mix, which is itself the operational lesson: telemetry thresholds are calibrated locally, not imported from a vendor datasheet.

Drawing failure-domain boundaries

A failure domain is the answer to a specific question: when this one thing breaks, what else breaks with it? Getting that boundary wrong in either direction is costly — draw it too large and a single faulty node stalls work across accelerators that had nothing to do with the fault; draw it too small and the isolation overhead (extra switching, extra redundant links, extra idle capacity) outweighs the protection it buys.

The scale of the problem is visible directly in Meta’s reliability study, which fits a failure model to their own operational data and reports observed mean time to failure of 47.7 days at an 8-GPU scale and 7.9 hours at a 1,024-GPU scale within their production clusters ^[3]. Beyond their observed range, the same fitted model projects MTTF continuing to fall — toward roughly 14 minutes at a hypothetical 131,072-GPU scale — which the authors present explicitly as a model projection rather than a measurement, since no cluster in their 11-month, 4-million-job, 150-million-A100-GPU-hour dataset actually ran at that size ^[3]. The same study found that while large jobs are the most failure-vulnerable, the majority of jobs in their clusters are small — over 90% of jobs use fewer than 8 GPUs — and that failures do not stay contained to the job they hit: roughly 16% of total lost “goodput” attributable to hardware failures came from secondary preemptions, where one job’s failure forced other, unrelated jobs off the cluster to make room for recovery ^[3]. That secondary-preemption effect is a failure-domain problem in a different guise — it is the scheduler’s isolation boundary failing, not the hardware’s.

Meta’s Llama 3 training run gives a concrete breakdown of what a real, large failure-domain surface looks like in practice: across 54 days of training on up to 16,000 H100 GPUs, the run experienced 466 total job interruptions — 47 planned and 419 unexpected — of which GPU-related issues accounted for 58.7% (faulty GPUs 30.1%, HBM3 memory failures 17.2%, other GPU component failures 12.7%), software-related dependency issues 12.9%, network infrastructure 8.4%, unplanned individual-host maintenance 7.6%, and suspected silent data corruption 1.4% ^[2]. Despite that volume of interruptions, the team reports needing manual intervention only three times across the entire period, with automation handling the rest — which is itself evidence that the failure domains and the automated recovery built around them were sized correctly for that fleet ^[2].

Hardware architecture can widen or narrow this boundary by design. Google’s TPU v4 system uses optical circuit switches to make interconnect topology dynamically reconfigurable rather than fixed at build time, a choice the paper frames as improving “scale, availability, utilization, modularity, deployment, security, power, and performance” simultaneously, and states that the optical switches themselves add under 5% to system cost and under 3% to system power while outperforming a fixed InfiniBand fabric on cost and power ^[11]. A reconfigurable fabric changes what a failure domain even means: instead of a fault permanently removing a fixed block of the machine, the topology can be redrawn around the faulty segment, shrinking the effective failure domain after the fact rather than only before it. That is a vendor engineering claim about one specific system, not a general property of optical interconnects, and it should be read as such — but it illustrates that failure-domain sizing is a decision made partly in hardware, not only in scheduling policy.

A scheduler for one fleet, two very different workloads

Training and inference make almost opposite demands on a scheduler. A training job is synchronous, long-running, and tolerant of being delayed as a whole; an inference request is short, latency-sensitive, and arrives continuously rather than in a scheduled block. Operating both on one shared pool of accelerators — increasingly the norm, since idle training capacity is expensive to leave unused and dedicated inference-only fleets are expensive to overprovision for peak demand — requires a scheduler that can reconcile both without starving either.

The empirical starting point is that GPU clusters do not behave like the general-purpose compute clusters that earlier schedulers were designed for. Microsoft’s analysis of the Philly cluster, a two-month trace of a large multi-tenant GPU deployment, found that GPUs behave as “a monolithic resource that cannot be shared at a fine granularity across users” and that deep learning’s requirement for gang scheduling — all of a job’s workers must start together — makes jobs “inelastic to failures” in a way that ordinary batch analytics jobs are not; the study used this to argue for scheduler designs purpose-built around locality and gang-scheduling constraints rather than adapted from general-purpose cluster managers ^[6]. Google’s own account of Borg, its long-running internal cluster manager, reports a workload with an extremely heavy-tailed resource distribution — the top 1% of jobs consume over 99% of cluster resources — alongside a multi-year trend of jobs migrating from a strict free tier into a best-effort batch tier and increasing use of resource overcommitment to keep utilisation high, changes the authors attribute partly to job dependencies driving a meaningful share of observed failures ^[9]. Both traces predate the current training/inference mix, but the structural lesson carries forward directly: a scheduler tuned for a uniform job population will misallocate a fleet where a handful of enormous synchronous jobs and a much larger population of small, latency-sensitive ones compete for the same hardware.

Purpose-built research schedulers have targeted this heterogeneity directly. Sia, presented at SOSP 2023, is described by its authors as a heterogeneity-aware, goodput-optimized scheduler that assigns elastic, resource-adaptive deep learning jobs across GPU types and counts, explicitly adapting its assignment as cluster load and job mix change over time rather than fixing an allocation at submission ^[12]. That elasticity is precisely the property a mixed training/inference fleet needs: an inference-serving allocation that can absorb a diurnal demand curve, and a training allocation that can be squeezed or expanded as spare capacity opens and closes around it. SenseTime’s Acme cluster offers a concrete example of scheduling two different workload phases on one system in production — their “decoupled scheduling for evaluation” separates GPU-bound model inference from CPU-bound metric computation, using a prior-based elastic round-robin allocation across sorted job queues, which the authors report reduced evaluation makespan by 1.3-to-1.8x compared to their prior coupled approach ^[4]. That evaluation workload is architecturally close to inference serving — bursty, latency-sensitive, competing with long-running training jobs for the same GPUs — which is why the same decoupling principle generalises to production inference scheduling on a shared training fleet.

The tuning knob every one of these systems exposes, in one form or another, is a priority weight between throughput-for-the-batch-job and latency-for-the-interactive-job, and none of the cited studies claim a universal setting for it. Sia’s own framing is that the correct assignment changes as the cluster’s load and job mix change ^[12]; Borg’s is that the tier boundaries themselves have shifted over the life of the system as workload composition changed ^[9]. The operational implication is that scheduler tuning is a recurring exercise tied to a fleet’s actual job mix, not a configuration set once during initial deployment and left alone — the same lesson the checkpoint-interval discussion above arrived at from a completely different direction.

Storage capacity planning for the training data pipeline

Storage for an AI training cluster has to serve two demand patterns that look nothing alike: a continuous, comparatively modest read load from the training-data pipeline feeding batches to the accelerators, and an intense, extremely bursty write load every time the fleet checkpoints. Sizing for the average of the two under-provisions for the burst; sizing purely for the burst wastes capacity the rest of the time. Meta’s account of the storage fabric built for Llama 3 pretraining makes the asymmetry explicit: their Tectonic-based storage fabric, spanning 240 petabytes across 7,500 SSD-equipped servers, was built to a sustainable throughput of 2 terabytes per second but a peak throughput of 7 terabytes per second — a more than threefold gap between the number that describes ordinary operation and the number that describes what checkpoint writes actually demand for a short window ^[2]. The paper describes this bursty checkpoint traffic as a central design challenge for the shared storage fabric, distinct from the steady-state demands of data loading ^[2].

MLCommons’s addition of a checkpointing workload to the MLPerf Storage v2.0 benchmark suite in 2025 is a direct response to operators needing a standardised way to size for that burst rather than guessing from a single vendor’s marketing throughput figure; the workload defines checkpoint tests across four model sizes — 8 billion, 70 billion, 405 billion, and 1 trillion parameters — and measures a storage system’s throughput specifically on the metric that matters operationally: since “a checkpoint save or load is only complete when every process is done,” overall performance is set by the slowest participating writer or reader, not the average one ^[8]. That single design choice in the benchmark — measuring the tail rather than the mean — mirrors the actual failure mode operators report: a storage array that looks well-provisioned on paper can still stall an entire fleet’s checkpoint if even one shard of it is slow.

The other lever, alongside raw provisioned throughput, is reducing how much has to move in the first place. Check-N-Run’s differential checkpointing and quantization techniques, described above for their effect on checkpoint interval, have an equally direct storage-capacity reading: a 2.5-to-8x reduction in required storage capacity for the same checkpointed model state is capacity an operator does not have to provision at all ^[5]. Acme’s asynchronous checkpointing pipeline, reporting up to a 58.7x reduction in checkpoint overhead on their larger tested model, similarly converts a storage-sizing problem into a smaller one before capacity planning even starts ^[4]. The practical planning sequence these accounts suggest, in order, is: first reduce checkpoint payload size and frequency using the techniques above, then size sustained throughput to the data pipeline’s steady-state read demand, and only then size peak throughput headroom to the resulting, already-reduced checkpoint burst — rather than sizing peak headroom to an unreduced checkpoint size and treating the resulting bill as fixed.

Common operational pitfalls

The published operator accounts converge on a similar list of pitfalls, distinct from the design questions above because they are things teams get wrong in execution rather than in policy.

Undercounting the initial hardware failure rate. Imbue reports that roughly 10% of their machines initially failed to boot at all, for reasons ranging from unconnected cables to broken power supplies and faulty NVMe drives, and that a further roughly 10% of their InfiniBand links showed elevated error rates before the fabric was fully brought up — problems the team describes as concentrated on a smaller set of “malcontent” nodes rather than spread evenly, and a steady-state expectation afterward that “about 3% of machines” would break in an average week ^[7]. Planning bring-up and steady-state capacity around a zero-failure assumption sets up a bad first month.

Treating intermittent faults as one-off noise. Imbue’s own operational conclusion, stated directly, is that “it’s worth writing tests and automated solutions for every kind of hardware or software failure you experience, since every issue encountered during training will reoccur” ^[7]. Gunawi and colleagues’ fail-slow study reaches the same conclusion from a different direction, by cataloguing how often what looks like a one-time glitch is actually a recurring fault mode that simply changed shape between occurrences ^[10].

Misdiagnosing network hangs as application bugs. Imbue describes NCCL collective-communication timeouts as especially hard to root-cause because a hang gives little indication of which node caused it, to the point that the team forked NCCL specifically to add better logging around timeout events ^[7]. Distinguishing a genuine software deadlock from a single slow or silently faulty network link on a large job is exactly the fail-slow diagnosis problem described earlier, applied to the network layer specifically.

Skewed job-size assumptions in optimisation priorities. Meta’s reliability study found that while large jobs are disproportionately likely to fail, over 90% of jobs in their clusters use fewer than 8 GPUs and collectively account for less than 10% of total GPU-hours — meaning an optimisation strategy built purely around protecting the largest, highest-visibility jobs can neglect the failure experience of the numerically dominant small-job population ^[3].

Background system activity synchronised badly across the fleet. Imbue traces periodic, unexplained throughput “sags” during distributed training to unsynchronised automatic garbage collection running independently on each host; disabling automatic collection and instead scheduling it synchronously across the fleet eliminated the pattern ^[7]. It is a small mechanism with a large, fleet-wide symptom, and it does not show up in any single node’s health check.

What is likely to standardise, and what would falsify it

These are forecasts, kept separate from the sourced findings above. Horizon: 15 August 2030.

One. Checkpoint intervals will increasingly be computed automatically from live, per-cluster MTTF telemetry rather than set once by a team and revisited only after an incident, following the logic MLCommons and Meta’s own reliability work both make explicit. Disconfirmed if, by the horizon date, published operator engineering accounts of large training clusters still predominantly describe a single static interval chosen at project start and unchanged through the run.

Two. Shared fleets serving both training and inference from one scheduler, rather than administratively and physically separate training and inference clusters, will become the default architecture at hyperscale, extending the elasticity direction visible in Sia and in Acme’s decoupled-scheduling work. Disconfirmed if the dominant hyperscale pattern by the horizon date remains dedicated, non-shared clusters per workload type.

Three. Fail-slow and silent-degradation detection will become a named, disclosed component of fleet telemetry stacks rather than an ad hoc addition each operator builds independently, given how consistently it appears as a distinct problem across the sources cited here. Disconfirmed if published fleet-telemetry descriptions in 2030 remain dominated by binary liveness checks with no stated methodology for detecting performance degradation short of failure.

Four. Failure-domain size will become a disclosed design parameter in cluster procurement and marketing, comparable to how power usage effectiveness became a standard disclosure for facility efficiency. Disconfirmed if procurement specifications and vendor system descriptions in 2030 continue to omit any stated failure-domain or blast-radius figure.

None of these four predictions requires a new hardware generation; each follows from operational pressure already visible in the sources above.

What to take away

Every decision covered here — checkpoint interval, telemetry threshold, failure-domain size, scheduler priority weight, storage headroom — has the same shape: it is not a constant an operator sets once, it is a function of the fleet’s current scale, current failure rate, and current job mix, and it has to be recomputed as those inputs move. Meta’s own numbers make the point starkest: mean time to failure measured at 47.7 days for an 8-GPU job fell to 7.9 hours at 1,024 GPUs in the same clusters ^[3], and no policy tuned for the first regime survives unchanged into the second. The teams whose engineering accounts this guide draws on did not solve reliability once at commissioning; they built the instruments — checkpointing systems, diagnosis pipelines, elastic schedulers, tiered storage — that let them keep re-solving it as the fleet grew out from under whatever assumptions the last policy was built on.