AI Accelerator Architecture in 2035: Scenarios, Signals, and Falsifiable Predictions

A ten-year architecture bet, made explicit

Ask a hardware architect where AI accelerators will sit in 2035 and most decline to answer, correctly, because the honest answer is a distribution over outcomes rather than a point estimate. This article builds that distribution anyway, in public, with the one discipline that makes a wrong guess worth more than a hedge: every claim below is tagged as a fact (measured or disclosed, and citable), a vendor claim (a company’s own assertion about its own product, reported as such and not endorsed), an analysis (a structural argument built from disclosed numbers), a scenario (one internally consistent path the evidence permits, not the one this article expects), or a prediction (a specific, dated, falsifiable claim with the observation that would prove it wrong stated in advance). Where people who study this professionally disagree, that disagreement is described rather than resolved in one direction. No scenario below is named as the likely one, and no cross-vendor ranking is built from any single vendor’s own reported speedup over its own prior product.

The angle is narrower than “will AI keep improving.” A companion piece in this publication works through what an accelerator physically is today — the matrix unit, the roofline model, HBM and advanced packaging, collectives, and current precision formats — in detail, and this piece does not repeat that account. What follows instead is five extrapolations: memory bandwidth per accelerator, packaging interconnect density, the narrowest numeric format in production use, the shape of the arithmetic array itself, and the physical limits that bound all three. Four of those are curves that have moved on a roughly exponential trajectory for a decade; the fifth asks what happens when at least one of them stops.

The documented present

Fact. JEDEC published the JESD270-4 HBM4 standard in April 2025, specifying transfer speeds of up to 8 Gb/s over a 2,048-bit interface for a total of up to 2 TB/s per stack — double HBM3’s per-stack bandwidth — with independent channels doubled from 16 to 32, each split into two pseudo-channels, and support for 4-high through 16-high stack configurations at 24 Gb or 32 Gb die densities ^[6]. The same standard set a deliberately relaxed nominal package thickness of 775 micrometres for both 12-high and 16-high stacks, a choice reporting on the standard describes as easing near-term pressure to adopt hybrid bonding for HBM stacking specifically, even as that same technique is already in volume use elsewhere in the package ^[6].

Fact and vendor claim. TSMC’s own account of its 3DFabric portfolio — SoIC, CoWoS, and InFO together — states that it lets customers “design their products more holistically as a system of mini-chips” rather than as one monolithic die, and frames advanced packaging as an integrated, end-to-end service rather than a step performed after the chip is finished ^[7]. That is the vendor’s own framing of why it built the service, not an independent measurement, and this article treats it as a claim about intent rather than a settled technical fact.

Analysis. Independent packaging analysis converges on a specific physical reason that framing exists: die size at the reticle limit — roughly 26 mm by 33 mm, or about 858 mm² — has bound the largest single die manufacturers can print for more than five years, and going bigger than one die means either accepting the yield loss of a larger monolithic part or splitting the design into chiplets and paying an interconnect and packaging cost to reassemble them ^[8]. The same analysis is explicit that chiplets are not a free win: designs run pad-limited as often as they run transistor-limited, because input/output pin density has scaled roughly three times more slowly than transistor density since 2006, so a smaller chiplet can end up needing proportionally more edge just to talk to its neighbours ^[8].

Fact. On the numeric side, NVIDIA’s Blackwell architecture introduced NVFP4, a 4-bit floating-point format (one sign bit, two exponent bits, one mantissa bit) using a two-level scaling scheme — a per-16-value block FP8 (E4M3) scale, plus a single per-tensor FP32 scale — that the company reports holds language-model degradation under roughly 1% relative to FP8 on tested workloads while cutting memory footprint by about 1.8 times versus FP8 and roughly 3.5 times versus FP16 ^[5]. A separate, broader industry effort produced the Open Compute Project’s Microscaling (MX) formats, standardising 8-, 6-, and 4-bit element types with a shared per-block scaling factor, with the underlying research reporting what its authors describe as the first demonstration of training a generative language model at sub-8-bit weights, activations, and gradients without changing the training recipe ^[4]. FP8 itself — E4M3 and E5M2 — was established two generations earlier as a format matching 16-bit training quality on models up to 175 billion parameters ^[3].

Fact. On architecture, the industry has not converged on one arithmetic template even at today’s scale. Google’s TPU v4 uses optically reconfigurable circuit switches to change interconnect topology in place, with the optical components reported at under 5% of system cost and under 3% of system power ^[2]. Cerebras builds a single wafer-scale die — the WSE-3, at 46,225 mm², roughly 4 trillion transistors, and 900,000 processing elements, each with its own local SRAM and router in a fine-grained dataflow mesh — that the company positions explicitly against multi-chip GPU clusters as “supercomputing performance with single-device simplicity” ^[9]. Groq’s LPU commits to a fully deterministic, compiler-scheduled dataflow design in which every instruction’s timing is fixed at compile time rather than resolved by hardware at run time, which the company states yields on-chip SRAM bandwidth “upwards of 80 terabytes per second” against roughly 8 terabytes per second of off-chip GPU memory bandwidth on comparable hardware ^[10]. These are three vendors’ own descriptions of three different bets on where determinism, memory locality, and reconfigurability should live in the arithmetic pipeline, not a ranking of which wins; none of the throughput figures above are comparable to one another because none were measured on the same workload under the same conditions.

Trajectory one: memory bandwidth

The structural fact behind every accelerator roadmap for a decade has been that peak arithmetic throughput has scaled roughly three times every two years while DRAM bandwidth has scaled roughly 1.6 times and interconnect bandwidth roughly 1.4 times over the same interval ^[1]. Analysis. Extending those three exponents unchanged from a 2024 baseline to 2035 — eleven years, or 5.5 doubling periods at the two-year cadence the source measures — gives a compact way to see how much the imbalance would compound if nothing structural intervenes:

With $r_F \approx 3.0$ and $r_B \approx 1.6$ for DRAM bandwidth, $(r_F/r_B)^{5.5} = (1.875)^{5.5} \approx 32$: the ratio of peak arithmetic to memory bandwidth would be roughly 32 times higher in 2035 than in 2024 if the historical exponents hold exactly. Swap in interconnect bandwidth’s 1.4x rate and the same arithmetic gives $(3.0/1.4)^{5.5} \approx 66$. Neither number is a forecast; both are what the stated assumption implies, and the assumption — that both exponents stay fixed for eleven more years — is exactly the thing worth doubting. HBM4’s own arrival already shows one departure from smooth continuation: doubling per-stack bandwidth required doubling channel count as well as pin rate, and the standard’s authors built in headroom (the relaxed 775-micrometre package thickness) specifically to avoid forcing hybrid bonding into HBM stacking before it is needed elsewhere ^[6] — a sign that bandwidth scaling and packaging scaling are coupled decisions, not independent curves.

Prediction. Horizon: end of 2030. Successive JEDEC HBM standards (HBM4E and any successor published before the horizon) will continue to raise per-stack bandwidth primarily through wider interfaces and more channels rather than through pin-rate increases alone, extending the pattern HBM4 already set ^[6]. Assumption: channel-count and interface-width scaling remain cheaper, in area and power, than pushing per-pin signalling rates further on the same electrical interconnect. Indicator: the per-pin data rate and total interface width disclosed in each successive published HBM standard. Disconfirmed if a published standard before the horizon achieves a majority of its bandwidth increase through per-pin rate alone, with interface width and channel count essentially unchanged from the prior generation.

Trajectory two: packaging density

Fact. TSMC’s SoIC hybrid-bonding process reached a 6-micrometre bond pitch in commercial-scale production in 2026, following a roadmap that industry reporting places at a further 4.5-micrometre pitch by 2029, with 3-micrometre prototypes already demonstrated ahead of that ^[7]. Analysis. That trajectory — bond pitch roughly halving every three to four years — is a packaging-density curve distinct from, but coupled to, the memory-bandwidth curve above: tighter bond pitch is what lets more independent channels and taller stacks fit in the same footprint, which is the physical enabler behind HBM4’s channel doubling and whatever comes after it ^[6]. But the reticle limit constrains this trajectory from a different direction. Independent packaging analysis notes that going to chiplets and multi-chip modules can yield real economic benefits — one detailed cost model puts the gain at roughly 30% more usable product per wafer once yield effects are included — but only once the interconnect and packaging costs of reassembling those chiplets are themselves cheap enough, and the same analysis is explicit that a poorly aligned chiplet split can raise total cost rather than lower it ^[8]. This is a place where practitioners genuinely disagree in emphasis rather than in fact: packaging vendors frame advanced integration as a designed-in system-level advantage ^[7], while independent cost modelling frames it as a constrained trade whose sign depends on how well a given chiplet split matches the underlying lithography economics ^[8]. Both are consistent with the same disclosed numbers; they differ on how much slack the industry has before the trade turns unfavourable again.

Prediction. Horizon: end of 2031. Hybrid-bonding pitch in commercial, high-volume packages will continue to shrink roughly on the pace already demonstrated — reaching a nominal pitch at or below 4.5 micrometres in volume production — without a change in the underlying bonding approach (still direct copper-to-copper hybrid bonding, not a substitute technique). Assumption: yield and cost per bonded interconnect continue improving fast enough to make the tighter pitch commercially attractive rather than merely technically demonstrable. Indicator: the bond pitch disclosed in TSMC’s, Samsung’s, or Intel’s published advanced-packaging roadmaps and volume-production announcements. Disconfirmed if volume-production hybrid bonding pitch is still above 5 micrometres at the horizon, or if a materially different bonding technique has displaced hybrid bonding as the leading approach.

Trajectory three: the precision-format floor

Fact. The move from FP8 to sub-8-bit formats has already happened once, and the mechanism by which it happened exposes the trade-off that will decide whether it happens again. A microscaling format with element width $w$ bits, block size $k$, and a shared scale of $s$ bits carries an effective per-element cost of

with the second term the amortised metadata tax. The OCP MX alliance’s MXFP4 uses $w = 4$, $k = 32$, and an 8-bit shared scale, for $w_{\mathrm{eff}} = 4.25$ bits per element — a roughly 6% tax ^[4]. NVIDIA’s NVFP4 instead uses a smaller block, $k = 16$, with the same 8-bit block scale plus a near-negligible per-tensor term, for $w_{\mathrm{eff}} \approx 4.5$ bits per element — a roughly 12.5% tax ^[5]. The smaller block buys better local adaptation to each block’s dynamic range, which is the stated reason NVIDIA gives for its accuracy results at 4-bit precision ^[5]; the price is paid in the overhead fraction, $s/(kw)$, which is exactly double NVFP4’s block-16 tax at block-32. Analysis. That relationship is the structural reason a further step to 2-bit or ternary elements is not simply “the same trick again”: at fixed block size and scale width, halving element width from 4 bits to 2 bits doubles the relative metadata tax, from 12.5% to 25% at $k = 16$. Holding the tax constant requires doubling block size to 32, which is the same move that cost MXFP4 its finer dynamic-range adaptation relative to NVFP4 in the first place. There is no free direction in this trade; every further step down in element width either accepts a growing metadata tax or accepts a coarser shared scale, and which one the industry chooses is not yet settled by any disclosed roadmap.

Prediction. Horizon: end of 2032. Production training stacks at frontier scale will still default to 4-bit or wider element formats (NVFP4, MXFP4, or a direct successor) rather than 2-bit or ternary formats, because the metadata-tax and dynamic-range trade above has not yet been shown to resolve favourably at frontier model scale. Assumption: no algorithmic technique emerges that decouples quantisation error from block size in the way outlier-aware or mixed-granularity scaling schemes have begun to attempt in research settings. Indicator: the default numeric format specified in major frontier-lab training-infrastructure disclosures and in NVIDIA’s and competitors’ flagship Tensor Core or matrix-unit specifications. Disconfirmed if a major lab discloses production frontier-scale training at 2-bit or ternary weights, activations, and gradients as a default rather than a research result, before the horizon.

Trajectory four: convergence or divergence

Analysis. The three architectures described in “the documented present” are not small variations on one template. A GPU’s streaming multiprocessors run independent, dynamically scheduled threads with a hardware-managed memory hierarchy; a TPU’s systolic matrix unit pumps operands through a fixed two-dimensional array with a software-managed on-chip buffer; Cerebras’s wafer-scale mesh distributes SRAM directly beside a million independent cores with no shared cache at all ^[9]; Groq’s LPU removes hardware scheduling entirely in favour of a compiler that fixes every instruction’s timing before the chip ever runs ^[10]. These four choices sit at genuinely different points in a design space, not at different price points on the same curve, and each vendor’s own framing of its choice claims a specific advantage: TPU v4’s reconfigurable optical topology is framed as adapting the network to the job rather than the job to a fixed network ^[2]; Cerebras’s framing is that removing the die boundary removes the interconnect cost of scaling out ^[9]; Groq’s framing is that removing runtime scheduling removes both a hardware cost and a source of latency variance ^[10].

Scenario. Two structurally different futures are consistent with this evidence, and the assignment of one architecture to a specific job — training at datacenter scale, latency-sensitive serving, or power-constrained edge inference — is what decides between them. Convergence is the case where workload requirements narrow enough, and the cost of maintaining a distinct hardware-software stack proves high enough, that one template absorbs the others’ best ideas: a GPU-shaped array gains a fully deterministic scheduling mode, or a systolic design gains a software-managed reconfigurable interconnect, until the meaningful competitive differences move into compilers and software rather than silicon topology. Divergence is the case where workload co-design keeps pulling architectures apart precisely because training, high-throughput serving, and edge inference have different bottleneck structures — training bound by interconnect and collective communication, low-latency serving bound by deterministic scheduling, edge bound by power per operation — and no single template optimises for all three well enough to justify abandoning the specialised ones.

Prediction. Horizon: end of 2032. At least two of the four architectural approaches disclosed above — general-purpose matrix-engine GPUs, systolic-array designs with software-managed on-chip memory, wafer-scale dataflow meshes, and fully deterministic compiler-scheduled dataflow processors — will remain in active commercial production and deployment at meaningful scale, rather than the market consolidating around one dominant template. Assumption: no single technique from one family (for example, deterministic scheduling, or wafer-scale integration) proves portable enough to absorb into the others without losing the property that made it distinct. Indicator: vendor product roadmaps and disclosed commercial deployments across at least two of the four families. Disconfirmed if, by the horizon, three or more of the four families have been discontinued, absorbed into a competitor’s architecture, or reduced to research-only status with no commercial shipping product.

Trajectory five: the physical limits that could force a discontinuity

Fact. Current-generation accelerator packages are already approaching a power-density ceiling that changes how they must be cooled. Reported single-GPU thermal design power has risen from roughly 300 watts (NVIDIA V100, 2017) to roughly 400 watts (A100, 2020) to roughly 700 watts (H100, 2022) to up to 1,200 watts (Blackwell-generation parts, 2024), with industry roadmaps cited in the same reporting anticipating chips drawing over 2,000 watts within one to two years and preparation already underway for 5-kilowatt-class chips; rack-level power density has risen roughly tenfold, from a historical baseline near 8 kilowatts per rack to around 100 kilowatts per rack in dense AI clusters ^[13]. That is a physical constraint on the package and the room around it, not on the die’s arithmetic, and it interacts with every trajectory above: denser packaging concentrates more power in less area, and higher per-pin memory bandwidth requires more power per bit moved unless the interconnect itself changes.

Vendor claim. NVIDIA has stated plans to move a substantial share of GPU-to-GPU and switch-to-accelerator communication onto silicon photonics with co-packaged optics, with reporting describing next-generation rack-scale platforms using optical engines placed directly beside switch and accelerator silicon, and specific switch products targeting throughput above 100 terabits per second per device ^[11]. Analysis. The stated rationale is a physical one: copper’s energy cost per bit rises with the distance a signal has to travel and the rate at which it must be driven, and at rising per-pin data rates the reach of a passive copper link at acceptable power shrinks toward the scale of a single rack rather than a full datacenter row. Moving photonics from the edge of the network into the package is a claim about which side of that trade-off — electrical reach versus optical conversion cost — has flipped, made by a company with a direct commercial interest in the answer, and this article reports it as exactly that: a stated plan, not yet a demonstrated industry-wide standard.

Analysis. IEEE’s long-range device and systems roadmap frames this kind of shift as expected rather than exceptional: it explicitly plans for heterogeneous and vertical monolithic integration to keep functional scaling going for at least another decade beyond conventional transistor scaling, treating packaging- and system-level integration as a first-class scaling lever rather than a stopgap ^[12]. Read against the reticle-limit and pad-limited-design constraints above ^[8], the roadmap’s own framing amounts to an institutional bet that the next decade’s gains come from integration choices — how dies, memory, and now light are combined — more than from any single transistor getting smaller.

Prediction. Horizon: end of 2030. Co-packaged optics will be in commercial production use for accelerator-to-switch or accelerator-to-accelerator links at the rack scale in at least one major vendor’s shipping product line, but will not yet have displaced copper-based links as the majority interconnect technology at the board level, within a single accelerator package. Assumption: the energy and manufacturing cost of optical-electrical conversion at the package edge falls fast enough to win at the rack boundary before it wins at the shorter, cheaper-to-serve board-level distances. Indicator: vendor product disclosures specifying the physical interconnect technology used at each tier of a shipping accelerator system, from board to rack. Disconfirmed if, by the horizon, either no major vendor has shipped co-packaged optics in production accelerator racks, or co-packaged optics has already become the majority interconnect at the board level inside individual accelerator packages.

Two axes, four scenarios toward 2035

Crossing the convergence-or-divergence question from trajectory four with the smooth-scaling-or-discontinuity question raised across trajectories one, two, three, and five gives four cells. The memory-bandwidth, packaging-density, and precision-format trajectories are not a third axis; they are the evidence stream that decides which side of the second axis the industry lands on, the same way a company’s funding durability can be a consequence of two more fundamental axes rather than an independent one. None of the four cells below is named as the likely outcome.

Scenario A: One Template, Smoothly Fed. A single dominant matrix-engine template absorbs the best ideas from systolic arrays, wafer-scale integration, and deterministic scheduling — software-configurable reconfigurable interconnect, optional compiler-scheduled execution modes, larger on-chip SRAM tiers — while HBM bandwidth, hybrid-bonding pitch, and precision formats all continue roughly the curves described above without a forced redesign. Specialisation increasingly lives in software and compiler stacks rather than in distinct silicon families. Horizon: a recognisably converged template by 2031-2032; smoothly continuing physical curves feeding it through 2035. Assumptions: the metadata-tax and reticle-limit constraints in trajectories two and three keep yielding to engineering rather than hardening into walls; no single specialised architecture’s advantage proves un-absorbable. Observable indicators: shrinking architectural diversity in new commercial accelerator announcements; continued incremental (not discontinuous) generational improvements in disclosed HBM, packaging, and precision-format specifications. Disconfirmation: falsified if, by 2032, three or more of the four architectural families from trajectory four remain independently commercially active, or if any of trajectories one, two, or three shows a sharp discontinuity rather than a continued curve.

Scenario B: One Template, Forced to Change Shape. Architectures converge on one broadly dominant template, but a physical wall — a pin-bandwidth plateau, a hybrid-bonding yield ceiling, or a precision floor reached earlier than trajectory three’s prediction assumes — forces that single template to absorb a genuinely new mechanism industry-wide at the same time, such as co-packaged optics becoming mandatory at the board level rather than only the rack level. Horizon: convergence recognisable by 2030-2031; a forced, simultaneous redesign event visible within a two-to-three-year window sometime before 2035. Assumptions: the dominant template’s specific physical bottleneck (whichever one binds first) is shared broadly enough across vendors that the forced change happens roughly together rather than staggered. Observable indicators: a sudden, coordinated shift in disclosed interconnect or packaging technology across multiple vendors’ flagship products within the same one-to-two-year window. Disconfirmation: falsified if convergence happens without any of the five trajectories showing a discontinuity, or if a forced change happens but architectures remain diverged rather than converged when it does.

Scenario C: Many Templates, Each Well Fed. Workload co-design keeps specialising silicon — training-optimised, latency-optimised serving, and power-optimised edge parts remain genuinely distinct families — while HBM bandwidth, packaging density, and precision formats all continue improving smoothly enough to feed every family without a shared crisis forcing consolidation. Horizon: continued architectural diversity recognisable through 2032; no shared discontinuity event through 2035. Assumptions: the economic cost of maintaining several distinct hardware-software stacks stays lower than the performance loss from forcing every workload onto one template; physical scaling curves have enough headroom left for all families through the horizon. Observable indicators: at least three of the four architectural families from trajectory four continuing to ship updated generations on roughly historical cadences. Disconfirmation: falsified if two or more previously distinct families merge into a shared architecture, or if a shared physical wall forces simultaneous redesign across the diverged landscape.

Scenario D: Many Templates, Each Hits Its Own Wall. Specialisation continues, but the physical limits in trajectory five bite different families at different times and in different ways — a wafer-scale design hits reticle- and yield-driven economics before a chiplet-based GPU design does, or a deterministic dataflow design hits an SRAM-density wall before a systolic array hits a pin-bandwidth wall. There is no single discontinuity event; instead a staggered series of forced, family-specific redesigns unfolds across an already-diverged landscape through 2035. Horizon: the first family-specific forced redesign recognisable by 2029-2030; a staggered pattern, rather than a single event, visible by 2035. Assumptions: the physical bottleneck differs meaningfully by architectural family rather than being shared; no single fix (such as co-packaged optics) turns out to relieve more than one family’s specific constraint at once. Observable indicators: forced architectural changes in vendor roadmaps occurring in different years for different families, each tied to a different disclosed physical constraint. Disconfirmation: falsified if a forced redesign event affects multiple architectural families simultaneously (which would indicate scenario B or a converged variant instead), or if no family shows a forced redesign at all through the horizon (which would indicate scenario C instead).

What survives across all four cells, and the wildcard none of them names

Three things hold regardless of which cell the evidence eventually favours. First, none of the five trajectories requires a single research breakthrough to keep moving; each has an incremental, engineering-driven path already visible in a disclosed roadmap or standard ^{[6, 7, 5]}. Second, the disagreement in trajectory two between packaging vendors’ own framing and independent cost analysis is a disagreement about magnitude and timing, not about the underlying physics — both sides agree the reticle limit and pad-limited-design constraint are real ^{[7, 8]}. Third, no scenario above requires any currently diverged architecture to be a mistake; TPU, Cerebras, and Groq’s own stated design rationales remain internally coherent claims about different bottleneck structures, not competing answers to the same question ^{[2, 9, 10]}.

The blind spot shared by all four scenarios is the same kind the roofline model itself cannot see: a workload-side shift large enough to change what “bound” even means. Every trajectory here assumes the current mix of dense matrix multiplication, attention, and collective communication remains the dominant computational pattern through 2035. A shift in model architecture profound enough to change that mix — sparse-by-default computation, a fundamentally different attention mechanism, or a training paradigm that needs far less interconnect relative to arithmetic — would move several of these curves at once, in a way none of the four scenarios traces, because all four assume the target the hardware is being built to hit stays roughly where it is today. That is a genuine limitation of a hardware-focused article, stated rather than hidden: these are scenarios for how the physical machine evolves given roughly its current job, not a claim that the job itself is fixed.

What to take away

A probe needle a hair’s width above a bond pad is not a measurement yet, and every number in this article’s five trajectories was, at some point, exactly that: a coupon in a tray, a card half-seated in a socket, a standard not yet ratified. Memory bandwidth, packaging density, and precision-format width have each compounded on something close to an exponential curve for a decade, and extending each curve mechanically to 2035 produces genuinely large numbers — a roughly 32-fold divergence between peak arithmetic and memory bandwidth, a hybrid-bonding pitch under five micrometres, formats with amortised metadata taxes below 15% at 4 bits. None of those numbers is a fact about 2035; each is what a stated, falsifiable assumption implies, with the observation that would prove it wrong written down in advance.

The more interesting open question is not any single curve but whether the industry’s architectural families converge or keep diverging while those curves play out, and that question is not answered by extrapolating harder. It is answered by which workloads keep needing genuinely different bottleneck structures — interconnect-bound training, latency-bound serving, power-bound edge inference — for long enough that maintaining separate silicon families keeps being worth it. This article’s four scenarios name that question honestly rather than resolving it in advance, because the record available in 2026 supports all four, and the discipline that matters is not picking the right one now. It is stating, for each, what a specific observation between now and 2035 would have to look like to prove it wrong.