The Hardest Unsolved Problems in Small and On-Device AI

The average is not where the failure lives

A model that fits inside a device’s memory, bandwidth and thermal budget is not thereby a solved problem. The physical envelope determines whether a model can run at all; it says nothing about whether the model that fits is reliable once it does. This article is about five places where that second question stays open — not because vendors have not tried, but because the research groups reporting progress also report, in their own words, where their method breaks down.

The discipline here is narrow on purpose. Every claim below traces to a specific finding a named study made about its own method, not to a general impression that small models “aren’t there yet.” Where a claim comes from a vendor rather than an independent or peer-reviewed source, it is marked as a vendor claim. Where researchers working on the same problem disagree about how serious it is, that disagreement is described rather than resolved in either direction. None of the five problems is a forecast that on-device AI will fail. Each is a boundary condition documented by the people building past it, and in every case the documenting is recent: the newest of the studies cited here is from February 2026, the oldest from 2022, and none of them treats its finding as closed.

The five: aggressive quantization can collapse accuracy on a specific task or a specific chip while the aggregate score barely moves; a device that learns from its user risks forgetting what it already knew; a quantization recipe tuned against one mobile accelerator does not reliably transfer to another; on-device fine-tuning marketed as private usually still depends on a server or a coordinating aggregator somewhere in the loop; and a headline compression ratio measures the case, not the capability inside it.

Problem one: the cliff a benchmark table cannot see

The standard way to report what quantization costs is a single number: average accuracy across a benchmark suite, before and after compression, expressed as a percentage retained. That number is not wrong. It is also not the number that determines whether a specific user’s specific request comes back right, because averaging is exactly the operation that hides a localized collapse.

Formally, if a model is evaluated on $T$ tasks with full-precision accuracy $\mathrm{Acc}_t^{FP}$ and quantized accuracy $\mathrm{Acc}_t^{Q}$ on task $t$, a benchmark table typically reports the mean regression

while what governs whether any individual deployment is safe to ship is closer to the worst-case regression

A quantization scheme can post a $\bar{\Delta}$ close to zero while $\Delta_{\max}$ is large, provided the loss concentrates on one or a few tasks that are a small share of the suite. Nothing about $\bar{\Delta}$ being small implies $\Delta_{\max}$ is small; the two only converge if degradation is spread evenly, and the studies below find that it is not.

A 2026 case study of post-training quantization for reasoning-focused LLMs deployed on Ascend NPU hardware reports exactly this pattern concentrated on one capability. The authors found that “aggressive 4-bit weight-activation schemes suffer from layer-wise calibration instability on the NPU, leading to logic collapse in long-context reasoning tasks,” while under the same conditions “standard 8-bit quantization remains numerically stable” ^[2]. Long-context reasoning is a small fraction of most benchmark suites by task count, so a suite-wide average would register this as a minor blemish even where, on the tasks that actually exercise it, the model’s answers have stopped being trustworthy.

The same shape of gap shows up when the axis is not a specific reasoning task but an entire mode of use. A 2025 study accepted at the International Conference on Machine Learning evaluated compressed language models not on perplexity or the standard natural-language-understanding suites but on agentic tasks — multi-step tool use, workflow generation, long-horizon planning — and found that “4-bit quantization preserves workflow generation and tool use (1%-3% drop) but degrades real-world application accuracy by 10%-15%” ^[3]. The paper’s own framing of the problem is direct: existing compression benchmarks “only focus on language modeling (e.g., perplexity) and natural language understanding tasks (e.g., GLUE accuracy), ignoring the agentic capabilities” that determine whether a compressed model is fit for the applications it is actually being compressed to serve ^[3]. A model can pass every standard quantization benchmark and still be materially worse at the one thing — acting reliably across several steps — that a shipping product needs it to do.

Neither finding is evidence that quantization is broadly unsafe; both papers report large, useful compression with genuinely small aggregate loss on the tasks their aggregate measures. The finding is narrower and harder to dismiss: the metric that is cheap to report (an average) and the metric that determines whether a specific deployment is safe (a worst case, or a capability the average never tested) are different quantities, and the gap between them is not visible unless someone specifically goes looking for it on the task or the hardware where it lives.

Problem two: personalization without erasing what came before

The commercial case for on-device models increasingly includes personalization — a model that adapts to one user’s writing, one household’s routines, one clinician’s shorthand, without shipping that data anywhere. Continual learning is the machine-learning name for updating a model on a stream of new data after deployment, and its oldest, best-documented failure mode is catastrophic forgetting: a model updated to do well on new data can lose competence on what it already did well, because gradient updates that improve performance on the new distribution are under no obligation to preserve performance on the old one.

A comprehensive 2024 survey of continual learning specifically for large language models frames this as an unresolved rather than a solved problem: pre-trained LLMs “tailored for specific needs, often experience significant performance degradation in previous knowledge domains — a phenomenon known as ‘catastrophic forgetting’” — and while the phenomenon itself has been “extensively studied” in the older continual-learning literature, the survey treats its behavior in LLMs specifically as presenting “new challenges” that the field has not settled ^[9]. That is a statement from the people mapping the field, not from a critic outside it.

One standard mitigation is regularization: penalize the optimizer for moving parameters that mattered to earlier tasks. The best-known form estimates a per-parameter importance weight — commonly the diagonal of the Fisher information, $F_i$ — from the old task, and adds it to the new loss:

where $\theta_i^{*}$ is the old optimum for parameter $i$ and $\lambda$ sets how strongly the old task is protected. This equation exposes the actual trade rather than resolving it: raising $\lambda$ protects old knowledge at the direct expense of how much the new update is allowed to change the model, and there is no value of $\lambda$ that removes the trade — only one that relocates it. It also exposes a cost specific to on-device deployment: computing and storing $F_i$ for every parameter, and doing so repeatedly as the device keeps learning, is itself memory and compute that a phone-class budget has to find room for, on top of whatever the update itself costs.

Whether continual learning methods actually hold up once they are moved onto the hardware that would run them is a separate, and separately unresolved, question. Hayes and Kanan’s study of online continual learning specifically for embedded devices notes a gap in the literature that predates the current wave of on-device language models: “several online continual learning models have been developed, [but] their effectiveness for embedded applications has not been rigorously studied,” and their own evaluation measures not just accuracy but “memory usage, compute requirements, and ability to generalize to out-of-domain inputs” precisely because prior work had largely skipped that accounting ^[10]. A comprehensive 2024 review of on-device language models lists adaptive learning and personalization among the paper’s identified “open challenges,” and frames closing them as requiring “interdisciplinary efforts” rather than an engineering fix already in hand ^[13].

The picture is not uniformly bleak, and the honest account has to include that. A 2025 comparative study of catastrophic forgetting across several open small models found meaningful variation by model family: on a set of continual fine-tuning runs over GLUE-style language tasks, “models such as Phi-3.5-mini exhibit minimal forgetting while maintaining strong learning capabilities,” with Orca-2-7B and Qwen2.5-7B also showing strong post-fine-tuning performance ^[11]. That is a genuinely different note from the survey’s framing of forgetting as a standing, general problem, and it matters for the disagreement discussed below: it is not settled whether catastrophic forgetting on-device is a property of the continual-learning problem in general, or something particular architectures and training recipes already resist better than others.

Problem three: one chip’s quantization is not another’s

A quantized model is usually described as though it were a single artifact — “the 4-bit version” — that behaves the same wherever it runs. It does not. The numerical behavior of low-bit inference depends on how a given accelerator’s compiler handles scaling, clipping, and operator fusion, and those choices differ by vendor and are frequently undocumented.

A 2025 study of cross-platform low-bit deployment on edge NPUs states the problem as bluntly as a methods paper allows itself to: “the same floating-point (FP) checkpoint can therefore yield inconsistent accuracy across backends, forcing practitioners to tweak flags or refactor models to vendor-friendly operator subsets,” because “vendor compilers differ in scaling, clipping, and kernel support, often as black boxes” ^[1]. Read plainly, this says that identical weights, quantized once, are not a portable object — the accuracy a team measures on the chip they validated against is not a reliable prediction of the accuracy the same file will get on a different vendor’s silicon, and closing that gap currently means retraining or hand-tuning per target rather than shipping one artifact everywhere.

The Ascend NPU case study cited above makes the same point from the hardware side rather than the compiler side: a quantization scheme’s stability is a property of the specific accelerator it runs on, not only of the bit width chosen, since the paper found instability specifically tied to “layer-wise calibration instability on the NPU” for a scheme that, on other hardware, is standard practice ^[2]. A recipe validated as safe on a GPU is not thereby validated as safe on a phone’s NPU; the failure mode is native to the chip it was not tested on.

This heterogeneity is compounded by mobile systems-on-chip rarely putting all of their available accelerators to work together. A 2025 study accepted at the ACM SIGOPS Symposium on Operating Systems Principles found that “existing designs typically only leverage a single AI accelerator for LLM inference, leading to suboptimal use of computational resources and memory bandwidth,” and built a system that draws on both a phone’s GPU and its NPU simultaneously to recover between “1.34x to 6.02x end-to-end speedup” left on the table by single-accelerator designs ^[12]. That is a genuine advance, but the size of the recovered gain is itself evidence of how much performance heterogeneous hardware was quietly losing under naive deployment before anyone measured it.

None of this is new in kind — cross-hardware inconsistency has been a known headache in mobile machine learning for years. The industry’s own standardized response, the MLPerf Mobile Inference Benchmark, exists specifically because ad hoc comparisons across device stacks were not trustworthy: the benchmark’s authors motivate the project by noting that their results “reveal the overwhelming complexity of the underlying mobile ML system stack, emphasizing the need for transparency in mobile ML performance analysis” ^[14]. A standardized benchmark that exists to make cross-device comparison legible is itself an admission that, without it, comparison was not legible — and MLPerf Mobile measures latency and accuracy on fixed models, not the specific interaction this article is about, between an aggressive quantization scheme and the one accelerator it happens to have been tuned against.

Problem four: fine-tuning that never leaves the device — almost

“On-device fine-tuning” is frequently offered as a privacy proposition: your data personalizes the model without ever leaving your phone. The engineering reality behind that phrase is more qualified than the phrase suggests, for a plain resource reason — backpropagation is far more memory- and compute-intensive than inference, and the memory budget that lets a small model run inference on a phone is usually not enough to also run its training.

MobiLLM, a 2025 system for enabling LLM fine-tuning on mobile devices, states the constraint its own design has to work around: “on-device LLM fine-tuning poses great challenges due to extremely high memory requirements and slow training speeds,” to the point that “resource-constrained mobile devices cannot afford” even parameter-efficient fine-tuning approaches without help ^[8]. MobiLLM’s answer is server-assisted side-tuning: the heavy backward pass is offloaded to a server while, in the authors’ words, “the data never leaves the mobile device” ^[8]. That is a real and useful privacy property — raw data does not cross the network — but it is not the same claim as “fine-tuning happens entirely on the device.” The state of the art for making mobile fine-tuning tractable at all currently routes around the device’s own limits by keeping a server in the loop for the part of the computation the phone cannot afford, which is a narrower and more conditional privacy guarantee than “on-device” alone implies.

Federated learning is the other common route to personalization without centralizing raw data: many devices train locally and only their model updates, not their data, are sent to a coordinating server for aggregation. DP-FedLoRA, a 2025 framework for privacy-enhanced federated fine-tuning of on-device LLMs, is explicit that this still needs its own safeguard, because federated fine-tuning on edge devices involves “processing sensitive, user-specific data, raising significant privacy concerns within the federated learning framework” even though raw data stays local ^[7]. Its response is differential privacy: each client “locally clips and perturbs its LoRA matrices using Gaussian noise” before anything is sent, with a formal $(\varepsilon, \delta)$-differential-privacy guarantee attached ^[7]. That noise is not free — it is a second trade-off, alongside the memory one, between how much personalization signal survives and how strong the privacy guarantee is.

Whether that guarantee holds up against a determined adversary is itself contested by newer results. A 2026 study introduced an attack — Projection Residual — specifically targeting federated large language models, and reported that it “achieves near 100% accuracy” at inferring whether a specific example was used in training, “outperforming prior methods by up to 75.75%” ^[6]. Most pointedly for the privacy claims above, the authors state that their attack “remains effective even under strong differential privacy defenses” ^[6] — the same category of defense DP-FedLoRA relies on. The paper’s own conclusion is not that differential privacy is worthless, but that the field’s threat models have been incomplete: the findings, in the authors’ words, “reveal a previously overlooked privacy vulnerability in FedLLMs and call for a re-examination of their security assumptions” ^[6]. Sending gradients or adapter weights instead of raw data is a meaningfully smaller attack surface than sending raw data — but “smaller attack surface” and “solved” are different claims, and the gap between them is precisely what this newer attack is measuring.

Problem five: what a compression ratio does not promise

A compression ratio is a statement about size: this many bytes became that many bytes. It is frequently reported alongside a retained-accuracy figure and treated, together, as a complete account of what was lost. Two recent findings suggest that account is routinely incomplete in ways that matter for what a compressed model is actually asked to do.

The agentic-capability study cited in problem one belongs here as much as there: the 10-15% real-world application accuracy drop the authors measured at 4-bit quantization occurred at a compression setting whose reported perplexity and GLUE-style scores looked fine ^[3]. A compression ratio and an aggregate accuracy figure, reported together, can both be accurate and still describe a model that has quietly lost the specific capability — sustained, multi-step, tool-using behavior — that an agentic product depends on.

Compression is also not linear in the way a single ratio implies. A 2026 structural pruning study of large vision-language models found a sharp inflection rather than a gradual slope: at 15% pruning, models “retain approximately 98-99% of their original capabilities,” and “even at a 30% compression ratio, the performance loss is minimal” — but “performance degradation accelerates significantly once the pruning ratio exceeds 30%,” and at 45% compression, retained performance for two of the evaluated models “drops to 83.99% and 86.22%, respectively, falling below [a] 90% fidelity threshold” ^[4]. The authors’ own stated boundary on their result is worth keeping attached to the finding: their recovery-training approach was demonstrated “at moderate compression ratios (up to 30%),” and beyond that point they state plainly that “performance loss becomes increasingly difficult to recover,” concluding that for more aggressive compression targets, “the extreme pruning of a large model is not a viable approach” using the methods they tested ^[4]. A single compression-ratio number cannot distinguish a model compressed at 25%, safely inside the flat part of that curve, from one compressed at 40%, past the point their own paper says its method stops working.

The remaining complication is that modern deployments rarely apply one compression technique in isolation — pruning, distillation and quantization are commonly stacked — and a 2025 systematic study of the order in which these techniques are applied found that the sequence itself changes the outcome, not just the total ratio achieved. The authors note that although pruning, quantization and distillation have each been “well studied” on their own, “their interactions and optimal sequencing remain unclear,” and that “the ordering of techniques significantly affects the final model quality” ^[5]. Concretely, they found that “applying quantization early causes severe performance degradation due to irreversible information loss that impairs subsequent training” ^[5] — meaning two teams reporting the same final compression ratio, using the same three techniques, can land at meaningfully different capability retention purely because of the order they applied them in, a variable a single ratio number does not record at all.

None of this means compression ratios are meaningless; they remain the right first-order measure of whether a model fits a device’s storage and bandwidth budget, which is a real and necessary question. It means a compression ratio answers a question about size and does not, by itself, answer the question of what the smaller model can still be trusted to do — and the tasks where that gap shows up are exactly the ones an aggregate benchmark, run at the wrong granularity, will not surface.

Where researchers disagree

Two disagreements run through the sources above, and neither is settled by simply picking a side.

The first is about how universal catastrophic forgetting actually is once model family and training recipe are held fixed. The continual-learning survey treats forgetting in LLMs as a standing structural problem worth a comprehensive review of mitigations ^[9]. A newer comparative study, evaluating specific small open models rather than the phenomenon in the abstract, found that some — Phi-3.5-mini specifically — showed “minimal forgetting” under continual fine-tuning where others did not ^[11]. Both findings can be true at once: forgetting may be a real and general risk while also being unevenly distributed across architectures and training recipes in ways the field has not yet explained. The disagreement is not about whether forgetting happens; it is about whether the right unit of analysis is “continual learning” as a general problem or specific model families as more and less resistant instances of it.

The second is about whether differential privacy, as currently deployed in federated on-device fine-tuning, is an adequate defense or a partial one whose adequacy depends on an attacker’s sophistication. DP-FedLoRA’s authors present their calibrated-noise mechanism as delivering “strong privacy guarantees” alongside competitive model performance ^[7]. The Projection Residual attack was built and tested specifically to probe that class of claim, and its authors report their method holds up “even under strong differential privacy defenses” ^[6]. This is not necessarily a contradiction — a formal $(\varepsilon,\delta)$ guarantee bounds a specific kind of information leakage under specific assumptions, and an attack can succeed by exploiting something the guarantee never covered. But it does mean a defense described as “strong” by its own authors and a documented attack that defeats it can both be accurately described, and a reader has to hold both rather than take either paper’s framing as the field’s final word.

Predictions, with the observations that would falsify them

These are forecasts, kept separate from the sourced findings above. Horizon: 12 August 2029. They assume no discontinuity in mobile accelerator design and no regulatory mandate forcing a specific privacy architecture.

One. Compression benchmarking will add a mandatory worst-task or minimum-retention figure alongside the aggregate score, because the aggregate has already been shown, on agentic and long-context tasks specifically, to hide the regression that determines shippability ^{[3, 2]}. Disconfirmed if the leading open compression leaderboards in 2029 still report only an aggregate retained-accuracy number with no per-task or worst-case figure.

Two. Quantization-aware calibration will become chip-specific practice rather than an artifact shipped once and assumed portable, because cross-backend inconsistency is a documented property of current compiler stacks, not a bug expected to disappear on its own ^[1]. Disconfirmed if a single quantized checkpoint format is validated as accuracy-equivalent across major NPU vendors by 2029 without per-vendor recalibration.

Three. On-device personalization products will increasingly disclose which parameters are frozen and which adapt, rather than describing “on-device learning” as a single undifferentiated capability, because the forgetting risk is now concentrated and documented rather than diffuse ^{[9, 10]}. Disconfirmed if major consumer platforms in 2029 still ship on-device personalization with no disclosure of what is protected from being overwritten.

Four. Differential-privacy claims attached to federated fine-tuning systems will increasingly be paired with an explicit threat model naming the attacks they defend against, rather than an unqualified “privacy-preserving” label, following direct demonstrations that at least one class of defense does not stop a documented attack ^[6]. Disconfirmed if federated fine-tuning products in 2029 still market differential privacy without specifying which attacks it is evaluated against.

None of these requires a breakthrough. They follow from documented gaps already on the record.

What to take away

Every problem above has the same shape: a convenient summary number exists, it is not wrong, and it is not the number that determines whether a specific deployment is safe. An aggregate accuracy score is not a worst-task score. A compression ratio is not a capability-retention guarantee. “On-device” is not automatically “fully local.” A differential-privacy bound is not automatically a defense against every attack built to test it. In each case the gap was not discovered by a critic arguing from first principles — it was measured and published by the same research communities building the techniques, in papers that name their own limitation plainly enough to quote.

That is the useful discipline for anyone evaluating a small or on-device model: ask which specific tasks were tested, on which specific hardware, at which specific compression setting, under which specific attack — and treat a claim that answers none of those as a claim still waiting for its measurement, not yet a finding.