A vocabulary problem before it is an engineering problem
Ten articles in this series will walk through grid interconnection studies, power-quality mitigation, liquid-cooling architectures, heat reuse schemes, water accounting, carbon accounting, and load-flexible operation, each assuming the reader already knows why an AI datacenter is a different kind of building from the one that housed a bank’s mainframe in 2005. This article supplies that foundation. It does not tour a working facility’s switchgear or its coolant distribution units — other installments in this series do that at the level of real equipment. Its job is narrower and comes first: why a cluster of accelerators concentrates so much more power into so much less space than a conventional server room ever did, what that concentration does to the two systems that have to absorb it — the grid connection and the cooling plant — and the handful of terms (kilowatts per rack, power usage effectiveness, water usage effectiveness) that later articles in this series will use without re-defining.
Start with the single fact that generates almost everything else: the accelerator chips that do AI training and inference dissipate far more heat per unit of silicon area, and far more heat per rack, than the general-purpose processors that filled datacenters for the previous two decades. Everything downstream — the liquid cooling, the busway sized for a rack instead of a row, the interconnection study that treats a single building like a small power plant, the concern about harmonics and oscillations at the substation — is a consequence of that one number moving.
Why the chip got hot
For decades, shrinking a transistor made it faster and lower power at the same time, a relationship formalized by Robert Dennard in 1974: as feature size fell, voltage and current fell with it, so power density — power dissipated per unit chip area — stayed roughly constant even as the chip did more work. Architects spent that dividend on clock speed for two decades. Dennard scaling stopped holding in the mid-2000s because transistor leakage and minimum operating voltage would not keep falling with feature size, and the industry’s answer was to stop chasing single-thread clock speed and put more cores on the same die instead. Esmaeilzadeh and colleagues showed why that second answer runs out too: modeling device scaling, single-core scaling, and multicore scaling together across projected process nodes, they found that at 22 nanometres — reached commercially within a few years of their 2011 paper — roughly 21% of a fixed-size chip’s area had to be left unpowered, or “dark,” to stay within a realistic power budget, a fraction that grows to more than 50% at 8 nanometres [8]. Silicon area had become cheaper to obtain than the power and heat-removal budget required to switch it all on at once.
An accelerator chip is the industry’s answer to that constraint. Instead of a few large, general-purpose cores switching on and off unpredictably, an accelerator packs thousands of small, simple arithmetic units that all perform the same operation on different data at the same time, and holds the whole array close to its power ceiling continuously rather than bursting to it occasionally. That design choice is exactly what makes the chip good at the dense matrix multiplications underneath a neural network, and it is exactly what makes it thermally punishing. A general-purpose server processor of the mid-2010s ran at a thermal design power in the range of 100 to 150 watts; NVIDIA’s H100 accelerator, the chip that trained a large share of the models put into production in 2023 and 2024, carries a documented maximum thermal design power of up to 700 watts in its SXM form factor [9]. That is not a modest increase spread across a larger fleet of chips — it is one package, in one socket, expected to shed nearly five times as much heat from very nearly the same die footprint.
Heat removal is fundamentally a problem of area, not of power alone. The quantity engineers actually design against is heat flux, power divided by the surface area it must leave through:
with
The multiplication does not stop at the package. NVIDIA’s current rack-scale system for accelerated computing arranges 72 of these GPUs, alongside 36 companion CPUs, across eighteen liquid-cooled compute trays inside a single cabinet, connected by a shared high-bandwidth interconnect [10]. Seventy-two packages, each individually pushing several hundred watts, now occupy the footprint a conventional rack design once assumed would hold perhaps forty general-purpose servers at a few hundred watts apiece between them. The rack, not the chip, is where this article’s central vocabulary problem becomes unavoidable.
The unit that broke: kilowatts per rack
A conventional datacenter rack — the sealed, floor-standing cabinet that a general-purpose cloud or enterprise workload has occupied for twenty years — is specified and sold by its power draw in kilowatts, because that number, not its physical volume, is what actually constrains how many of them a room can hold. A rack drawing five to ten kilowatts was unremarkable for most of the 2010s; the room’s air handlers, its raised floor, and its aisle containment were all engineered around a fairly narrow band of watts per square metre of floor. Accelerated computing broke that band. The International Energy Agency’s technical analysis is explicit about the mechanism, not only the outcome: “the rise of AI is accelerating the deployment of high-performance accelerated servers, leading to greater power density in data centres,” and it projects that electricity consumption by accelerated servers specifically will grow at roughly 30% a year in its base case, against roughly 9% a year for conventional servers occupying the very same facilities [2]. Two different growth rates inside the same building means two different kinds of rack are now sharing one roof, and the industry’s own accounting shows the accelerated share pulling away. Lawrence Berkeley National Laboratory’s most recent national estimate finds that total US datacenter electricity demand “more than double[d] between 2017 and 2023” on the strength of accelerated-server growth, with the sector’s compound annual growth rate accelerating from roughly 7% before 2018 to roughly 18% between 2018 and 2023 [3].
None of that growth is spread evenly across racks; it is concentrated in the specific cabinets that hold accelerators, which is why “kilowatts per rack,” not aggregate site megawatts and not floor area, is the basic unit of density this series will keep returning to. A rack at legacy density and a rack at current AI density are no longer the same kind of object merely scaled up. A rack drawing on the order of a hundred kilowatts or more is closer, electrically, to a small dedicated substation than to a taller version of a five-kilowatt cabinet, and it has to be fed, protected, and cooled as one.
Grid interconnection, from first principles
Every one of those dense racks eventually draws its power from a point where a private electrical system meets the public grid — the interconnection. Interconnection is not a cable; it is a regulatory and engineering process. A prospective large customer, or a generator wanting to sell power, submits a request to the transmission owner or grid operator, who must study whether the surrounding network — the substations, transformers, and transmission lines already in place — can absorb the new load or generation without violating reliability limits, and what upgrades would be required if it cannot. Only after that study is satisfied, and often only after the customer commits to paying for identified upgrades, is a connection agreement issued. Because thousands of projects request this at once and studies must be worked through in some sequence, requests queue, and the queue itself is now a primary constraint on where an AI datacenter can be built and when it can begin drawing power — a point the IEA makes plainly: unless grid-connection risk is addressed, “around 20% of planned data centre projects could be at risk of delays,” and separately, “wait times for critical grid components such as transformers and cables have doubled in the past three years” [1].
What makes an AI datacenter a distinct case within that process, rather than simply a larger version of an ordinary large customer, is defined by the organization that actually operates the North American bulk power system. NERC’s Large Loads Task Force, formed because reliability coordinators were seeing a wave of unfamiliar interconnection requests, characterizes an emerging large load as “any commercial or industrial individual load facility or aggregation of load facilities at a single site behind one or more point(s) of interconnection that can pose reliability risks to the BPS due to its demand, operational characteristics, or other factors” [4]. That definition is deliberately not just about size. A steel mill and an AI training cluster can draw the same peak megawatts and still pose very different risks to the grid that feeds them, because what matters for interconnection planning is not only how much power a load takes but how it takes it — steadily, or in the coordinated steps a training cluster can produce, which is the subject of the next section.
Power quality: what a datacenter actually does to the wire it is plugged into
“Power quality” is the general name for everything about delivered current and voltage besides its average level: whether the voltage stays sinusoidal, whether it holds steady moment to moment, and whether the load draws current smoothly or in sharp discontinuous pulses. A conventional building’s load — offices, lighting, ordinary computing — is close enough to smooth and linear that a utility can treat it as a well-behaved statistical average. A large AI training cluster is neither smooth nor especially well described by an average, and NERC’s white paper is unusually direct about why: “depending on the operating performance, large loads can have relatively low energy consumption while idling and then have sudden demand spikes when expected to operate,” and “the extensive use of power electronics-based devices could make data centers a significant source of harmonics, unless filtering is designed to address those harmonics” [4]. Harmonics are currents or voltages at multiples of the grid’s fundamental frequency, produced whenever equipment — uninterruptible power supplies, variable-speed cooling fans, the switch-mode power conversion inside every server — draws current in anything other than a smooth sine wave. Left uncorrected, they distort the voltage waveform for every other customer sharing the same feeder; NERC’s documented example shows voltage distortion at a real datacenter falling from roughly 6% to well under 1% of the fundamental once harmonic mitigation was installed [4].
The other half of the problem is timing rather than waveform shape. A large training job advances in synchronized steps — computation, then a collective communication step where thousands of accelerators exchange data together, then computation again — and because the accelerators draw power together, that computational rhythm becomes an electrical rhythm. NERC’s own field measurement of an AI training cluster’s demand over a two-minute window shows real power cycling repeatedly between roughly 60% and 100% of its peak several times a minute, a pattern the report describes as “periodic, repetitive, and sustained in nature” [4]. At small scale that is a curiosity; at the scale of an interconnection-sized load, a repeating disturbance at exactly the wrong frequency can excite a “forced oscillation” — sympathetic vibration in generators, turbines, or the network itself — and the report documents a real 2023 case in which a Midwestern datacenter’s power electronics inadvertently produced a one-hertz forced oscillation that propagated into the surrounding grid before settling out [4]. None of this makes a training cluster dangerous by itself; it makes it a load type the bulk power system was not originally characterized to handle, which is exactly why NERC assembled a task force for it rather than filing it under ordinary industrial load.
Why liquid cooling stopped being a choice
Return to the heat-flux relationship from earlier and ask where it runs out of room. Air is a poor working fluid for carrying heat away from a hot surface — its ability to absorb heat per unit of volume moved is roughly three orders of magnitude below that of liquid water, which is why every high-power industrial process that predates computing at all, from engines to furnaces to power plants, eventually settled on a liquid or phase-change loop rather than a fan. Datacenters avoided that conclusion for decades only because their chips stayed under a heat-flux ceiling that a sufficiently large volume of moving air could still manage, at the cost of considerable fan power and duct space. ASHRAE’s Technical Committee 9.9, the body that publishes the industry’s thermal guidelines, dates the end of that arrangement precisely: “over the last decade the industry had a period where significant performance increases were delivered, generation over generation, accompanied by modest and predictable power increases. That period ended around 2018. Large power increases in the compute, memory, and storage subsystems of current and future IT equipment are already challenging data centers… Liquid cooling is becoming a requirement in some cases, and should be strongly and quickly considered” [7].
The mechanism ASHRAE identifies is not simply that chips got hotter; it is that the case temperature a chip can tolerate has been falling even as its power has been rising, a genuinely unwelcome combination. The case temperature, the measurable point on top of the package that a cooling system actually has to hold at or below its limit, stands in for a junction temperature deep inside the silicon that cannot be measured directly. As transistors have packed closer together and switched faster, chip vendors have had to lower the maximum case temperature they will certify even as the power flowing through that same case has climbed, and ASHRAE warns plainly that “with case temperatures decreasing in the future, it will become increasingly harder to use higher ASHRAE classes of both air and water” [7]. A cooling medium is being asked to remove more heat and hold a lower temperature ceiling at the same time, and air cooling, already close to its practical limit, has no second lever left to pull. Direct-to-chip liquid cooling — a cold plate mounted straight onto the package, with coolant carrying heat away at the point where the flux is highest and the available surface is smallest — closes most of that gap by using a fluid whose volumetric heat capacity is roughly 3,600 times that of air, so the same heat can be carried away by a modest flow through a compact channel instead of a large volume of moving air through a duct.
PUE and WUE, precisely
Two ratios recur throughout this series, and both are simpler than their reputation suggests. Power usage effectiveness compares the total electricity a facility draws to the electricity that reaches the computing equipment itself:
The Green Grid, the industry consortium that introduced the metric in 2007 and later consolidated its guidance into a single reference document, defines it plainly: “PUE for a dedicated building is the total facility energy divided by the IT equipment energy. PUE is an end-user metric used to help improve energy efficiency in data center operations” [5]. An ideal value is 1.0, meaning every watt drawn by the site reaches the computing load; anything above 1.0 is overhead — cooling, power conversion losses, lighting, and the rest. The same document states the metric’s limit as clearly as its definition: “PUE is not a data center productivity metric, nor is it a standalone, comprehensive efficiency metric. PUE measures the relationship between the total facility energy consumed and the IT equipment energy consumed” [5]. It says nothing about whether the computing being done is useful, and nothing about water.
Water usage effectiveness fills that second gap, introduced by the same organization a few years before this decade’s AI buildout made it urgent:
reported in litres per kilowatt-hour and deliberately built to use “the same value… as the denominator for this new metric” as PUE, “to both ensure the metrics stay linked and speed their adoption” [6]. WUE exists because the cheapest way to reject a given amount of heat is very often evaporative — cooling towers and adiabatic assistance work against the wet-bulb temperature rather than the dry-bulb temperature, so they can reach a lower effective cooling temperature for less fan and compressor energy than a fully dry system — and that efficiency is paid for in water rather than electricity. A facility can improve its PUE by switching from dry to evaporative rejection while its WUE gets worse in the same stroke, which is exactly why the two ratios are reported together rather than treated as substitutes for one another.
What this leaves for the rest of the series
The vocabulary above is deliberately narrow. It says nothing yet about how a cluster reuses its rejected heat, how a site accounts for the carbon embedded in the electricity it buys, how flexible or inflexible a training job’s schedule really is against grid conditions, or how a specific building’s structural floor loading and water rights actually constrain where any of this can be built — each of those is a full article elsewhere in this series, built on the physical facts established here. What should already be visible is that none of those later questions are independent of one another. A site chosen for cheap, abundant power runs into the interconnection queue described above before it runs into anything about cooling. A cooling architecture chosen to minimize PUE can push WUE past what a water-stressed site can sustain. A load reshaped to ease the power-quality signature described above is, by definition, a load with less freedom to chase the cheapest electricity hour. The physical constraints this article introduces do not resolve independently; they trade against one another, and the rest of this series is substantially an account of those trades.
Predictions, with the observations that would falsify them
These are forecasts, separated from the sourced analysis above. Horizon: 15 August 2029.
One. Per-rack power ratings for new-build AI capacity will be specified and marketed the way voltage class and cooling type already are — a headline figure buyers compare directly — rather than left implicit in floor-area or aggregate-megawatt totals. Disconfirmed if 2029 capacity announcements still report only site-level megawatts without rack-level power figures.
Two. Grid operators will formalize a distinct interconnection and ride-through category for large, periodic power-electronics loads, separate from steady industrial load, extending the direction NERC’s own task force has already taken. Disconfirmed if by 2029 NERC or FERC still treat large computing loads under the same standards as constant industrial load, with no additional power-quality study requirement.
Three. WUE will be reported alongside PUE as a matched pair in the majority of new hyperscale sustainability disclosures, rather than as an occasional supplementary figure. Disconfirmed if PUE remains the dominant or sole disclosed efficiency metric in majority practice by 2029.
Four. Air-cooled accelerator products will become a minority of new AI compute shipments within the horizon, as case-temperature and heat-flux limits continue to tighten. Disconfirmed if a majority of newly shipped AI accelerators in 2029 are still specified for air-only cooling.
None of these requires a technology discontinuity. Each follows from constraints already visible in the sourced material above.
What to take away
An AI datacenter is not a bigger version of the datacenter that came before it; it is a different kind of electrical and thermal object, and the difference starts at the chip. A package that must shed several times the heat of its predecessor from very nearly the same area forces a rack to be measured in kilowatts rather than furniture, forces the grid connection to be treated as its own engineering study rather than an afterthought, and forces the cooling loop to move liquid rather than air past the point where any amount of moving air could keep up. Read the density, the interconnection queue, the power-quality signature, and the PUE/WUE pair together, and the rest of this series’ subjects — heat reuse, carbon accounting, water accounting, load flexibility — become elaborations on a single starting fact rather than separate stories.
