Two axes, not four separate bets
Ask four people building small models where on-device AI will be in 2035 and the answers cluster into four seemingly independent bets. Will dedicated mobile neural processing hardware close the raw-compute gap with the accelerators a frontier model trains and serves on — the way Qualcomm’s own Hexagon NPU rating nearly doubled in about a year and a half, from “up to 45 TOPS of theoretical performance” in the original Snapdragon X Elite to an “80 TOPS Hexagon NPU” in the Snapdragon X2 Elite [5, 4] — or does a persistent hardware gap, the kind visible the instant a single rack-scale training system is set beside a phone’s silicon, keep cloud dependency structurally necessary for the hardest everyday tasks? Will on-device personalization graduate from swapping in a small pre-trained adapter to a model that keeps learning from a person’s own use without quietly forgetting what it already knew — the exact problem a 2024 survey of continual learning in large language models still calls an open, “non-trivial challenge” [7] — or does it stay confined to prompt-level context management, the pragmatic choice a 2026 paper on on-device agent memory makes deliberately because that memory “is hard-bounded and exposed” and “becomes an attack surface” the moment it is treated as durable [8]? And will privacy-preserving on-device processing become the default a user simply expects, the way Google already states for its on-device model that the system “doesn’t store any record of the input data or the resulting outputs after processing” [10] — or does Apple’s own admission of why it built a server tier at all, that “to process more sophisticated requests, Apple Intelligence needs to be able to enlist help from larger, more complex models in the cloud” [11], describe a boundary hardware progress will not erase?
Argued one at a time, these read as four independent coin flips. They are not. The first two are the same underlying question approached from two directions — the hardware substrate and the workload it is asked to carry — and the second two are also the same underlying question approached from two directions: how a model is shaped to one person, and how much of that shaping is allowed to leave the device. This article uses two axes instead of four. Axis A, capability, asks whether the compute gap between on-device hardware and cloud-frontier hardware closes far enough that most everyday tasks stop needing a cloud round-trip, or whether it persists. Axis B, personalization-privacy, asks whether on-device continual learning becomes reliable enough to serve as the working basis for privacy-preserving AI by default, or whether personalization stays limited to what a prompt and a context window can hold, which keeps a cloud-hybrid architecture necessary regardless of what the hardware alone can do. Crossing them gives four scenarios, each with a horizon, assumptions, observable indicators, and an explicit disconfirmation condition. None is named as the likely outcome.
The documented present
Fact. Mobile NPU throughput has moved fast by its own standard. Qualcomm’s Snapdragon X Elite, benchmarked in mid-2024, carried a Hexagon NPU rated at “up to 45 TOPS of theoretical performance” [5]. Its 2025-announced, 2026-shipping successor, the Snapdragon X2 Elite, carries “an 80 TOPS NPU,” reported as delivering “37 percent higher performance and 16 percent lower power consumption compared to its predecessor” [4]. That is real, vendor-disclosed progress inside roughly eighteen months.
Fact. Set beside cloud-frontier hardware, the same numbers look very different. Nvidia’s own specification page for the GB200 NVL72 — a 72-GPU, rack-scale system, not a single chip, and the comparison below is offered as a scale illustration rather than a like-for-like ratio — states “720 POPS” of INT8 Tensor Core throughput and “13.4 TB HBM3E” of GPU memory at “576 TB/s” of bandwidth across the rack [3]. A single mobile NPU’s 80 TOPS is roughly four orders of magnitude below that rack’s aggregate INT8 figure. Nothing about Axis A depends on that gap closing to zero — it depends on whether it closes far enough, for enough everyday tasks, that the difference stops being decisive.
Fact. The gap that matters in practice is not always raw TOPS. A 2025 paper on mobile-NPU test-time compute found that “mobile NPUs have underutilized computational resources during typical LLM inference,” and built a hardware-aware quantization and lookup-table scheme specifically to spend that idle headroom on parallel test-time scaling, reporting that “smaller models using test-time scaling can match or exceed the accuracy of larger models, achieving a new performance-cost Pareto frontier” [2]. That is evidence for the optimistic pole of Axis A: some of the apparent gap is currently left on the table by software rather than fixed by silicon.
Fact. The same 2024 survey that frames the field’s constraints also states its current limits in blunt, quantified terms: “executing a state-of-the-art 405-billion parameter model on a smartphone would be unfeasible without substantial compromises,” a 70-billion-parameter model is “impractical” on “a personal office laptop,” and a 7-billion-parameter model’s inference “will consume about 0.7J per token,” which on “an iPhone with a battery capacity of about 50kJ” caps a continuous conversation “at most” around two hours [1]. Those are 2024 figures on 2024 hardware and will move, but they anchor how far Axis A’s optimistic pole currently has to travel.
Fact. Apple’s own account of its on-device foundation model captures the tension inside Axis A in one place. The on-device model, “~3 billion parameters,” is reported to outperform “larger models including Phi-3-mini, Mistral-7B, Gemma-7B, and Llama-3-8B” — real evidence that a small, efficient model can beat larger open ones on general capability. But Apple’s separate server-side model, built for exactly the requests the on-device model cannot handle, “compares favorably to DBRX-Instruct, Mixtral-8x22B, GPT-3.5, and Llama-3-70B” [6] — the same document is evidence for both poles of Axis A at once: small models have gotten much better, and a frontier-adjacent server tier still exists because some requests still need it.
Fact. Google’s Gemma 3n line reports an architectural move in the same direction as Apple’s, aimed at the memory wall rather than the raw-compute one: models with “raw parameter count” of “5B and 8B” are engineered to “run with a memory footprint comparable to traditional 2B and 4B models, operating with as little as 2GB (E2B) and 3GB (E4B) of memory” [13]. That is a vendor claim about a vendor architecture, offered as one, but the direction is consistent with the test-time-compute paper above: part of what looks like a hardware gap is actually an engineering gap that software and architecture keep narrowing without waiting for new silicon.
Fact. On the personalization side, the mechanism shipping today is the adapter, not continual learning of the base model. Apple describes using “adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks,” with “the parameters for a rank 16 adapter typically” requiring “10s of megabytes” [6]. An adapter is cheap, swappable, and reversible — and it is also, definitionally, not the base model learning from a person’s ongoing use; it is a small pre-trained module selected or shipped for a task.
Fact. The idea of training directly from on-device use without centralizing the data is a decade old, not new. Google’s original federated learning announcement describes phones that “collaboratively learn a shared prediction model while keeping all the training data on device,” where “the improved model on your phone can also be used immediately, powering experiences personalized by the way you use your phone” [12]. That the foundational technique is nearly ten years old and personalization still mostly ships as static adapters rather than continuously updating weights is itself evidence about how hard the harder version of the problem has turned out to be.
Fact. The harder version has a name and an unresolved status in the current literature: catastrophic forgetting. The 2024 survey states it plainly — “pre-trained LLMs, when tailored for specific needs, often experience significant performance degradation in previous knowledge domains,” and integrating them into “dynamic data distributions, task structures, and user preferences” remains “the non-trivial challenge” the field is still organizing itself around [7]. A 2026 paper on on-device agent memory responds to the same problem by refusing to update weights at all, instead governing what an agent remembers with “a single net-value-per-byte score” that decides what to “KEEP,” “SHARE,” or discard, explicitly because unmanaged on-device memory “consumes RAM and energy” and “becomes an attack surface” [8]. Two different 2024-2026 papers, two different architectures, one shared conclusion: durable on-device learning is treated as a risk to be governed, not a capability to be trusted outright.
Fact. On the privacy side, the vendor claims are specific and architectural rather than merely promissory. Google states that its on-device AICore service “doesn’t store any record of the input data or the resulting outputs after processing” and “does not have direct internet access,” routing any network traffic through “the open-source Private Compute Services companion APK” [10]. Apple’s Private Cloud Compute documentation makes the comparable claim for its server tier rather than avoiding the cloud altogether — data handled there benefits from the fact that “Apple retains no privileged access” and “runtime transparency is cryptographically assured through Secure Boot” [11]. Both are vendor claims about vendor systems, reported here as claims; the architectural difference between “the data never left the device” and “the data left the device under an audited, attested boundary” is real regardless of whose claim is credited.
Fact. Consumer trust in the alternative — sending data to the cloud and hoping it is handled well — is moving in a direction that could push demand toward on-device defaults. Deloitte’s 2025 survey of roughly 3,500 US consumers found that concern about “data privacy and security jumped from 60% to 70%” year over year, that only “20% of surveyed respondents say tech providers are ‘very clear’ about what data they collect or how it’s used,” and that just “27%” report “‘high’ or ‘very high’ trust that tech providers are keeping their data secure” [9]. That is evidence about appetite for the privacy-by-default pole of Axis B; it is not, by itself, evidence that the underlying continual-learning problem gets solved fast enough to satisfy that appetite on-device.
Why these two axes, and why the default computing substrate is a consequence, not a third
Axis A is consequential because it decides what an on-device model is even asked to be: a genuine substitute for a cloud request on most everyday tasks, or a fast front end that still hands off anything demanding to a server. It is uncertain because the same document can be read for both poles at once — a 3-billion-parameter model beating much larger open competitors, and a server tier built for exactly the requests that small model still cannot handle [6].
Axis B is consequential because it decides whether privacy-preserving on-device AI is a property of the deployment or a promise about how the cloud tier is operated. It is uncertain for a structural reason visible in the 2024-2026 literature: the technique that would make on-device personalization deep rather than shallow — continuously updating the base model’s own weights from a person’s use — is the technique two independent recent papers treat as too fragile or too risky to do casually, preferring adapters and governed memory instead [7, 8].
Whether on-device AI becomes the default computing substrate for everyday tasks — the thing a phone or a laptop does locally as a matter of course, with the cloud as the exception rather than the rule — is not a third axis; it is what the other two jointly produce, and testing that jointness is exactly what the room’s validation bench exists to do. Write
Four scenarios toward 2035
Crossing the two axes gives four cells. None is named as the likely outcome; each is named for the object in this article’s own room that best captures its logic.
Scenario one: The Rails Meet — capability converges, personalization and privacy converge
Mechanism. Mobile and embedded NPU throughput keeps compounding at something like the pace the Hexagon line has already shown [5, 4], while software techniques such as test-time compute on underused NPU headroom keep buying additional effective capability without waiting for new silicon [2]. In parallel, the continual-learning problem the 2024 survey calls unresolved gets solved well enough — through better regularization, better on-device memory governance, or a successor to today’s adapter pattern — that personalization becomes durable rather than swapped-in, and privacy-by-default becomes a natural consequence of not needing to send data out to get a good answer.
Horizon. Recognisable capability convergence for most everyday tasks by 2031-2032; reliable, trusted on-device personalization at consumer scale plausible by 2034-2035.
Assumptions. The Qualcomm-style hardware trajectory continues rather than flattening; the catastrophic-forgetting problem the 2024 survey documents proves to be an engineering gap rather than a durable property of how these models are trained.
Observable indicators. Flagship on-device models stop needing a documented server-tier hand-off for the majority of everyday requests, the way Apple’s own model split still requires today [6]; on-device continual-learning methods graduate from research papers explicitly built to avoid weight updates [8] to shipped defaults that do update weights safely.
Disconfirmation. Falsified if, by 2032, flagship consumer devices still route a large share of routine requests to a cloud tier, or if on-device personalization is still delivered exclusively through swappable pre-trained adapters rather than continuous learning from use.
Scenario two: The Bench Outruns the Rack — capability converges, personalization and privacy stall
Mechanism. The hardware and raw-capability trajectory keeps compounding — mobile NPUs keep gaining the way the Hexagon line already has, and small models keep closing the gap the way Apple’s 3B model already closes it against much larger open competitors [6] — but the continual-learning and memory-governance problems the 2024-2026 literature documents turn out to be durable rather than transitional [7, 8]. Devices become fast and broadly capable without becoming deeply personal, so meaningful personalization keeps requiring either a periodic cloud-trained adapter update or continued reliance on server-side context, keeping a cloud-hybrid architecture in place for reasons that have nothing to do with raw compute.
Horizon. Recognisable capability convergence by 2030-2031; a personalization-privacy gap that persists through 2035 without necessarily closing.
Assumptions. Raw capability and durable personalization are more separable than Scenario One’s optimistic pole assumes; investment concentrates on the hardware and quantization problem, which is closer to solved, faster than on the harder continual-learning problem.
Observable indicators. On-device models handle demanding everyday tasks locally and fast, yet personalization keeps being delivered through periodically refreshed adapters or cloud-side profile data rather than continuous local learning; vendors keep publishing adapter-based personalization architectures [6] rather than shipping deployed continual learning.
Disconfirmation. Falsified if durable, weight-level on-device personalization becomes a shipped default within the horizon — that would indicate the world has moved to Scenario One instead.
Scenario three: The Rack Outruns the Bench — capability stalls, personalization and privacy converge
Mechanism. The continual-learning and memory-governance problems get solved — on-device models learn reliably and safely from a person’s own use, and privacy-by-default becomes real for what the device can do — but the raw hardware gap illustrated by the NPU-versus-rack contrast above does not close for the hardest everyday tasks [3, 1]. The result is a device that knows its user deeply and keeps that knowledge local, while still handing off demanding reasoning, generation, or search tasks to a cloud tier — Apple’s existing on-device/server split [6, 11], but with the local half doing far more of the personalizing work than it does today.
Horizon. Recognisable personalization-privacy convergence by 2030-2031; a capability gap that persists through 2035 for a defined class of demanding tasks.
Assumptions. Continual learning is fundamentally a data-governance and algorithmic problem rather than a compute problem, so it can be solved on today’s hardware trajectory; the hardware gap illustrated by rack-scale accelerators is structural enough that it does not close within the horizon even as NPUs keep improving.
Observable indicators. Consumer devices ship default continual personalization with data never leaving the device, while system documentation continues to disclose a defined, non-shrinking class of tasks still routed to a cloud tier; vendors report privacy-by-default as achieved while still maintaining and expanding server-side capability tiers.
Disconfirmation. Falsified if the class of tasks requiring cloud escalation shrinks to a small residual within the horizon — that would indicate the world has moved to Scenario One instead.
Scenario four: The Rails Stay Apart — capability stalls, personalization and privacy stall
Mechanism. Neither pressure resolves. The hardware gap the GB200-versus-NPU contrast illustrates keeps mattering for a wide range of everyday tasks, not just the hardest ones, and the continual-learning problem the 2024 survey and the 2026 memory-governance paper both treat as unresolved stays unresolved [3, 7, 8]. Cloud-hybrid architectures remain dominant for the reasons already visible today: cost and capability both favor a server tier for anything beyond routine, low-stakes tasks, and personalization keeps happening mostly through prompt-level context and periodically shipped adapters rather than anything that could be called learning. This scenario’s leading indicator is not a future event; it is the documented present above, continuing.
Horizon. Close to today’s baseline; recognisable as the stable case by 2029 if none of the trends above accelerate; could persist through 2035.
Assumptions. No architectural or algorithmic breakthrough forces convergence on either axis; the economics that currently favor a hybrid split — routine, low-stakes tasks locally, demanding or personalization-heavy tasks routed to a cloud tier with its own privacy architecture [11] — continue to hold as compute and model-training costs move together rather than apart.
Observable indicators. Published on-device capability figures keep trailing cloud-frontier figures by a wide, stated margin; personalization keeps shipping as swappable adapters [6] rather than continuous learning; consumer trust in cloud data handling stays low, as the Deloitte figures already show [9], without materially changing where the processing actually happens.
Disconfirmation. Falsified if either capability convergence or personalization-privacy convergence is observed at the thresholds defined in Scenarios One through Three — either observation would move the world out of this cell.
What all four share, and the wildcard neither axis names
Three things hold across every cell. First, a cloud tier of some kind survives in three of the four scenarios and only fully disappears from everyday use in the one — The Rails Meet — where both axes resolve toward the optimistic pole simultaneously; even Apple’s own architecture, arguably the most aggressive current bet on the device, keeps a server tier specifically because “more sophisticated requests” need it [11]. Second, which specific technique wins inside each axis’s optimistic pole — a successor to today’s LoRA-style adapters, a regularization method that finally tames catastrophic forgetting, a new NPU architecture rather than an incremental TOPS increase — is a narrower, more replaceable engineering detail than the axis itself; the axis is compatible with any winner. Third, none of the four requires a plateau in frontier cloud capability. A cloud-frontier model can keep improving on the hardest tasks while on-device capability keeps closing the gap on everyday ones, because
The four scenarios share a blind spot too. All four assume roughly the gradual movement the documented present already shows. A large-enough shock would not fit cleanly into any of them. A single training or memory-architecture technique that solved catastrophic forgetting the way the 2024 survey frames it — turning continual learning from a risk to be governed into a capability to be trusted [7] — could move Axis B abruptly rather than along any of the four scenarios’ gradual paths, and would likely drag Axis A’s economics with it, since a device that can learn durably from its user becomes worth far more silicon budget than one that cannot. Equally, a single, widely publicized failure of on-device memory security — of the kind the 2026 agent-memory paper already treats as a live risk rather than a hypothetical one [8] — landing in a shipped consumer product rather than a research benchmark could set both personalization and trust back further and faster than The Rails Stay Apart’s gradual baseline implies. Either event would move both axes at once, abruptly, rather than along the paths each scenario traces on its own.
Two predictions, stated separately from the scenarios
Prediction one. Horizon: end of 2029. At least one major mobile or laptop chipmaker will report a generation-over-generation NPU throughput increase of less than 30%, breaking the roughly doubling-every-two-generations pattern the Hexagon line has shown from 45 to 80 TOPS [5, 4]. Assumption: the easy architectural gains — wider vector units, better quantization support — get exhausted before the memory-bandwidth and power-density limits that constrain the whole class of mobile silicon are engineered around. Indicator: official vendor NPU TOPS disclosures at each major product-line refresh. Disconfirmed if the major mobile chipmakers’ flagship NPU ratings continue compounding at or above the rate observed from 2024 to 2026 through the end of 2029.
Prediction two. Horizon: end of 2031. No major consumer AI platform will ship on-device continual weight updates — as opposed to adapter selection, retrieval, or context management — as a default, non-opt-in feature for its general-purpose assistant. Assumption: the risk profile the 2026 on-device memory paper describes, where unmanaged local learning “becomes an attack surface,” continues to outweigh the personalization benefit for a mass-market default, even as research prototypes improve [8]. Indicator: vendor documentation and developer platform releases describing how on-device personalization is implemented. Disconfirmed if a major platform ships default continual weight-level learning from user interaction, with the update happening on-device, by the horizon.
Neither prediction requires a capability discontinuity. Both follow from the structure already visible: two genuinely uncertain axes, a present documented well enough to anchor a horizon, and a field that has only recently started separating “the model got faster” from “the model learned something about you and kept it local” — the exact distinction that would let anyone outside the labs tell which cell of the matrix the world is actually in.
What to take away
The refusal to pick a favorite among these four cells is the substantive claim, not a hedge around one. In mid-2026 the evidence is genuinely split on both axes at once: a mobile NPU line that nearly doubled its rated throughput in eighteen months sits beside a single rack-scale cloud system whose own disclosed numbers are still roughly four orders of magnitude larger [5, 4, 3]. A 3-billion-parameter on-device model that beats much larger open competitors sits beside the same vendor’s own admission that a separate, larger server tier exists precisely because some requests still need it [6, 11]. A decade-old technique for keeping training data on the device sits beside a 2024 survey stating that the harder version of the same idea — continuous learning without forgetting — remains a “non-trivial challenge,” and a 2026 paper that responds to that challenge by refusing to update weights at all [12, 7, 8]. Anyone reporting a single confident future for small and on-device AI in 2035 is reporting which of these four cells they would bet on, not what the current record shows.
The more useful discipline is the one this article tried to practice: name the axis, ground each pole in a dated figure or a vendor’s own stated limits, and say in advance what observation would mean the world had moved to a different cell. Small and on-device AI has already demonstrated real things — mobile NPUs approaching triple-digit TOPS, small models beating much larger ones on general benchmarks, a decade of federated and adapter-based personalization technique. Whether any of that closes into the reliable, private, largely self-sufficient everyday computer its more confident advocates already describe it as becoming is still open, and it will stay open until both the tracking wall and the adapter rack in this article’s own room report a reading, not just one of them.
