Ten Failure Modes That Define Small and On-Device AI Deployments

Ten ways a deployed model differs from a benched one

A model that has passed every evaluation on a training cluster is a model that has been tested exactly once, under conditions its maker controlled completely: a single warm run, room temperature, mains power, one hardware target, and a benchmark suite designed to summarize behaviour into a small number of scores. Deployment onto a fleet of small and on-device models is not one more evaluation of the same kind. It is a sustained condition — hours of runtime, real thermal mass, a battery instead of a wall socket, hundreds of silicon variants, months of drift across a fleet that was never one population to begin with. Each of those differences opens its own failure mode, and each one is invisible from the bench that produced the score everyone cites.

This piece works through ten of them in the rough order a device would surface them: first the model itself under aggressive compression, then the physical envelope it runs inside, then the fleet it belongs to, then the claims made about it. Each failure mode is anchored to a documented study or a reported case rather than a plausible-sounding guess, and each is a distinct mechanism — fixing one does nothing for the other nine.

1. The accuracy cliff the aggregate benchmark dilutes to nothing

The standard argument for shipping a four-bit or eight-bit quantized model is an aggregate benchmark score that barely moves. That argument is true and it is also the wrong instrument for the question that matters in production, because an aggregate score is a weighted average across many subtasks, and a real cliff on one narrow slice can vanish inside it.

Marchisio and colleagues quantified exactly this gap for multilingual generation. Automatic metrics reported only modest average degradation from quantization, but on Japanese specifically “a 1.7% average drop in Japanese across automatic tasks corresponds to a 16.0% drop reported by human evaluators on realistic prompts” ^[1]. The paper’s broader finding is that non-Latin-script and lower-resource languages are disparately affected by quantization, and that the automatic metrics used to justify shipping a compressed model systematically understate the harm on exactly the tasks where it lands hardest. Ten times the visible degradation, hiding inside a headline number that looked fine.

The mechanism generalizes beyond language. Any capability that is a small fraction of a pretraining and evaluation mix — a minority language, a rare tool-call format, an uncommon unit system, a niche domain vocabulary — occupies a small fraction of the aggregate score’s weight, so a compression step that damages it disproportionately barely moves the number a release decision is based on. The number that clears a release gate and the number that describes what a specific user experiences are not the same number, and quantization is one of the few compression steps applied uniformly to a model regardless of which capabilities are load-bearing for which users.

2. Thermal throttling that only shows up after the bench run ends

A benchmark run is short. A conversation, a transcription session, or a camera pipeline is not, and the difference matters because mobile silicon is thermally limited in a way a benchmark’s first ten seconds never exercises.

Laskaridis and colleagues built an automated infrastructure to benchmark language models across real mobile and edge hardware and ran sustained sequences of prompts rather than single-shot measurements. On an iPhone 14 Pro running a 3-billion-parameter model at 4-bit precision, they observed throughput “dropping with two bumps happening on the 20th and 32nd prompts on average,” which they attribute to the device shifting into different energy and dynamic-voltage-and-frequency-scaling states as it heats — with device temperature reaching 47.9°C over the course of one sustained conversation ^[2]. A 50-watt Jetson AGX edge device run for comparison showed a smooth, flat throughput line under the same workload. Both devices ran the identical model. Only the phone had a skin-temperature limit and no fan.

The practical consequence is that a benchmark number reported at the first prompt is not a prediction of the twentieth. Any evaluation of a battery-powered device that stops before a governor has had reason to intervene is measuring a state the device will not be in for most of its actual use.

3. Out-of-memory crashes when the free memory was never contiguous

A device can report gigabytes of free memory and still kill the process that asks for a much smaller allocation, because the operating system’s out-of-memory killer does not care about total free memory — it cares about whether a single contiguous block large enough for the request exists. Large weight tensors and growing key-value caches on a constrained device fragment memory into scattered free regions faster than they fill it, and an allocation can fail against a device that is, in aggregate, nowhere near full.

is the condition that actually governs whether an allocation succeeds, and it can fail even when $S_{\mathrm{request}} \le \sum_i \mathrm{free}_i$ holds comfortably — the requested size fits inside total free memory but not inside any one free block. This is precisely the problem the smartphone inference system PowerInfer-2 was built to manage: it introduces a segmented neuron cache with a least-recently-used eviction policy specifically to keep the working set of “hot” and “cold” neuron clusters from fragmenting a mobile device’s limited DRAM, allowing models “whose sizes exceed the device’s memory capacity” to run without the allocator failures that naive loading produces, while cutting memory usage by roughly 40% relative to prior mobile inference engines at comparable speed ^[3]. The fix is not more free memory; it is memory management that treats fragmentation as the binding constraint rather than total capacity.

4. Silent numerical drift across accelerator vendors and driver stacks

The same trained weights, given the same input, do not reliably produce the same output across different hardware — and the divergence is silent, because nothing crashes and no error is raised. Schlögl, Hofer and Böhme measured this directly across 75 distinct inference platforms and found that “despite using a fixed trained model and fixed input data, inference results are not consistent across platforms, and sometimes not even deterministic on the same platform,” tracing the main causes to differences in SIMD instruction use on CPUs and the runtime selection of convolution algorithms on GPUs ^[4]. That study covered CPU and GPU backends specifically; the mechanism it identifies — vendor-specific kernel implementations that reassociate the same floating-point sum in a different order — is architecturally present, if anything more acutely, on mobile NPU accelerator stacks, where each vendor ships a private compiler, a private operator set, and its own quantization scheme, with no shared reference implementation across chip generations.

For a fleet of devices spanning several NPU vendors and driver versions, the consequence is that “the model” is not one function. It is a family of slightly different functions, one per accelerator stack, and the differences concentrate on inputs near a decision boundary — exactly where a wrong answer is most likely to matter and least likely to be caught by a spot-check on easy cases.

5. Catastrophic forgetting after on-device personalization

On-device personalization — adapting a small model to one user’s writing style, vocabulary, or usage pattern through local fine-tuning — is a form of continual learning, and continual learning has a well-established failure mode: adapting to new data degrades performance on what the model already knew.

Luo and colleagues ran a systematic empirical study of this effect across large language models from 1 billion to 7 billion parameters undergoing continual instruction tuning, evaluating forgetting across domain knowledge, reasoning, and reading comprehension. Their central finding is unambiguous: “catastrophic forgetting is generally observed in LLMs” across that entire size range, with severity that tends to increase with scale rather than decrease ^[5]. They also identify a partial mitigation — general instruction tuning before the domain-specific pass reduces the degree of forgetting — but a mitigation is not an elimination.

The relevant point for on-device personalization specifically is that the failure is not hypothetical or exotic; it is the default outcome of naive continual fine-tuning, observed generally across model families and scales. A personalization pipeline that periodically fine-tunes a local adapter on a user’s recent activity, without an explicit mechanism to preserve prior capability — replay of earlier data, a frozen base with an isolated adapter, or a regularization term penalizing drift from the original weights — should be assumed to be quietly eroding capabilities the user never asked to trade away, capabilities that were part of what shipped as “the model” in the first place.

6. Battery drain from an always-on path that was never power-budgeted

An always-on feature — a wake word, a background music identifier, a live-translation detector — is only cheap if it was architected to be cheap, and the architecture that makes it cheap is not the obvious one. Apple’s own account of the “Hey Siri” detector describes a staged system precisely to avoid this trap: a small, low-power auxiliary processor runs a compact acoustic model continuously, and only escalates to the full detector — and eventually the main application processor — when it believes it has heard the trigger phrase, specifically “to avoid running the main processor all day just to listen for the trigger phrase” ^[12]. That staging is the entire design, and it exists because the alternative — running any meaningfully capable model continuously on the main compute path — is not a viable power budget for a battery-powered device.

The failure mode is what happens when a product ships an always-on capability without that staging discipline. Reporting on always-on AI features across recent Android releases catalogues a specific list of features that keep the NPU or GPU active in the background well beyond their apparent scope — ambient music identification, live translation running continuously across messaging and media, and generative wallpaper effects that keep the GPU rendering as the phone moves — and describes the mechanism plainly: features that “chew through RAM, constantly call home to cloud servers, and keep the NPU firing all day” ^[9]. The gate that is supposed to be a five-milliwatt listener becomes, without careful budgeting, a background process running at a meaningful fraction of full compute, continuously, for a benefit the user experiences only occasionally.

7. Cold-start latency a warm-model benchmark deletes

A server keeps its weights resident in accelerator memory across millions of requests, so the cost of loading them is amortized to nothing and every benchmark implicitly measures a warm model. A device frequently does not have that luxury: the model may not be resident when the user taps, and the first token has to wait on weights being paged in from flash storage.

Alizadeh and colleagues addressed this directly, keeping the bulk of a model’s parameters in flash and streaming only what is needed per token, using windowing to reuse activated neurons across consecutive tokens and row-column bundling to read larger, more efficient chunks from storage; the result was models “up to twice the size of the available DRAM” running with inference “4-5x faster” on CPU and “20-25x faster” on GPU than naive on-demand loading ^[6]. PowerInfer-2 pushed the same problem further with finer-grained neuron-cluster loading and pipelining, reporting an average speedup of 25.4x, and up to 29.2x in the best case, over the two prior state-of-the-art frameworks it compared against, including the flash-loading approach above ^[3]. Both results are impressive engineering and both make the same point by existing: absent this work, a cold on-device model pays a storage-bandwidth tax that a benchmark measuring only steady-state tokens-per-second never reports, because steady state is defined as the condition after that tax has already been paid.

8. Model-version drift across a fleet that was never one population

“The fleet” implies a single population running a single model. It rarely is one. Devices check in at different times, on different connectivity, holding whatever build happened to be cached when they last successfully updated, and a fleet of any real size is therefore always a mixture of model versions rather than one version answering every request.

Bonawitz and colleagues’ account of Google’s production federated learning system for mobile devices is built around exactly this reality: the system has to coordinate model training and updates across a population of devices that are heterogeneous by construction — intermittently connected, only eligible to participate under specific conditions such as being idle, charging, and on an unmetered connection — rather than assuming a synchronized, uniformly up-to-date fleet ^[7]. That paper addresses training coordination specifically, but the same structural fact governs any fleet of deployed models: a device that has been offline, on a metered connection, or simply unlucky in update scheduling is still serving requests, on an older cached build, indistinguishable from its neighbours from the outside.

The operational consequence is that “what did the model say” is not a well-posed question for a fleet at any given moment without also asking which build answered. A regression fixed in the newest release can still be live in production for weeks on devices that have not yet successfully pulled it, and a support ticket referencing “the current behaviour” may describe a build that was superseded a month earlier everywhere except on the one device in front of the user filing it.

9. Privacy claims a fallback cloud call quietly breaks

“Runs on-device” and “never leaves the device” are architectural claims, and architectural claims are falsified by a single exception, not by a pattern. A model that processes ninety-nine requests locally and forwards the hundredth to a server for capacity, capability, or convenience reasons has not partially kept its privacy promise — for that hundredth request, the promise was false, and the user who was told the feature was private has no way to know which request that was.

The gap between claim and practice has been measured directly. NowSecure scanned roughly 183,000 mobile apps in 2025 and found 18% used AI in some form, while 2% were observed actively sending data to remote AI endpoints — with some of that traffic transmitted unencrypted to cloud-based AI services ^[8]. The reporting on always-on Android AI features cited above notes the same fallback pattern from the product side: capabilities that a device cannot handle locally are quietly escalated to a cloud service as a matter of course, which is a sound engineering decision and simultaneously the exact mechanism that breaks an unqualified on-device claim ^[9]. The precedent for what an undisclosed fallback costs in trust is not hypothetical: Amazon’s Alexa voice assistant was reported in 2019 to route a portion of user recordings to a global team of human reviewers, processing roughly a thousand audio clips per shift, a practice users had not been clearly informed of and had no obvious way to opt out of before it became public ^[10]. Alexa was never marketed as strictly on-device, so the comparison is not one-to-one — but it is the clearest documented case of the general pattern: a system whose actual data path diverges from its perceived one, discovered by users only after the fact, is a trust failure independent of whether the underlying processing decision was reasonable.

10. Numerical instability at the extreme low-bit edge

Below a certain bit width, quantization stops being a uniform, gentle loss of precision and starts being an occasional, sharp failure concentrated on specific values rather than spread evenly across the weight distribution. Dettmers and colleagues documented the mechanism precisely for 8-bit quantization at scale: as transformer language models grow past a certain size, “systematic emergent features” — outlier activation dimensions of unusually large magnitude — appear and come to dominate attention and prediction quality, and standard quantization schemes that treat all values uniformly fail specifically on these outliers rather than failing uniformly ^[11]. Their fix, isolating the outlier dimensions into a separate 16-bit computation while quantizing the remaining 99.9% of values normally, is itself evidence for the diagnosis: a uniform low-bit scheme was numerically unstable precisely where the outliers lived, and no amount of averaged accuracy across a benchmark suite would have surfaced that as a distinct, localized failure rather than a general precision cost.

At the more extreme end of the compression spectrum common in on-device deployment — three and four bits rather than eight — the same class of problem does not disappear; it becomes more consequential, because there is less headroom to absorb an outlier without saturating the representable range entirely. A weight or activation that overflows a 4-bit representation’s dynamic range does not degrade gracefully; it clips or wraps, and the resulting error is not proportional to the precision lost — it can be arbitrarily large on the specific input that happened to trigger it, while leaving every other input untouched and every aggregate metric unmoved.

Ordering the ten by how they actually surface

None of these failure modes require an adversary or a bug in the ordinary sense. Each is what a correctly implemented system does under conditions a bench does not reproduce. Sequenced by how they tend to appear in a fielded product: a quantization cliff on a specific task is present from the day of release and simply undetected; thermal throttling and battery drain accumulate over minutes to hours of a single session; memory fragmentation and cold starts are per-request risks that scale with how long a device has been running other things; NPU numerical drift and fleet version skew are properties of which specific unit answered a given request; catastrophic forgetting accumulates over the weeks a personalization pipeline keeps adapting; and privacy-claim failures and low-bit instability are typically discovered only when someone goes looking for the specific case that breaks them — which is precisely why they tend to surface latest and cost the most in trust when they do.

Predictions, with the observations that would falsify them

These are forecasts, separated from the documented findings above. Horizon: 12 August 2028.

One. On-device model evaluation will add a mandatory sustained-load and thermal-state axis to standard reporting, because a first-prompt benchmark number will have become recognized as structurally unable to predict tenth-minute behaviour. Disconfirmed if leading on-device model cards in 2028 still report only single-shot or warm-state throughput with no duration or thermal disclosure.

Two. Quantization evaluation for models intended for deployment will shift from a single aggregate score toward mandatory disaggregated reporting by task and language subgroup, driven by cases like the multilingual cliff documented here. Disconfirmed if release decisions for major on-device model families in 2028 are still gated on a single blended benchmark number with no subgroup breakdown published.

Three. Fleet-version observability — knowing which cached model build actually answered a given request — will become a standard field in on-device telemetry, because the alternative (debugging behaviour without knowing which of several live builds produced it) will have proven too costly at scale. Disconfirmed if major on-device AI platforms in 2028 still provide no per-request build attribution to support engineers.

Four. Public trust in “on-device” and “private by design” claims will increasingly require an independently auditable network-silence guarantee rather than a policy statement, following the pattern set by transparency reports for server-side processing. Disconfirmed if by 2028 no major platform has shipped a verifiable, inspectable claim of this kind for any on-device AI feature.

None of these four requires a capability breakthrough. They follow from treating deployment as the sustained condition it is rather than the single event a benchmark measures.

What none of this fixes

Better compression will not fix fleet-version skew. Better memory management will not fix a catastrophic-forgetting personalization loop. A verified network-silence guarantee will not fix a thermal throttling curve. The ten failure modes above do not share a root cause, and there is no single architectural decision — not federated learning, not quantization-aware training, not a stricter privacy review, not a bigger battery — that closes more than two or three of them at once. Anyone shipping a small or on-device model who believes one initiative will retire this list has not yet separated the model from the device from the fleet from the claim made about all three, and each of those four things fails on its own schedule, for its own reason, discovered by whoever happens to be using the one unit that hit it first.

What to take away

A model that scores well is a model that has cleared one test, once, under conditions someone else chose. A model that keeps working is a model whose maker has separately verified it against sustained thermal load, fragmented memory, accelerator diversity, continual adaptation, an unbudgeted always-on path, a cold cache, a heterogeneous fleet, an audited data path, and the specific low-bit inputs that do not average away. None of the ten checks substitutes for another, and skipping any one of them means finding out about it from a user instead of a bench.