The gap between a demo and a fleet
A model that answers correctly on the phone sitting on an engineer’s desk has cleared the lowest bar in the entire project. Whatever fixed envelope of memory, memory bandwidth, sustained thermal headroom, battery energy per query and cold-start latency that device presents, the demo has shown the model fits inside it — on that one device, today, with nobody else’s app also asking for the display, the radio and the thermal budget at the same time.
Passing that bar answers only whether the model can run. It says nothing about whether the feature built around it keeps working once it leaves the one phone it was built on: whether the compression choice made for that phone’s accelerator is even the right choice for the fleet the feature actually ships to; whether the app knows what to do when the local model is asked something outside its competence; whether the ten-thousandth device in that fleet, not the review unit, meets the assumptions the demo quietly made; whether a model shipped six months from now reaches users in a way that does not fracture the install base into incompatible cohorts; and whether the privacy line on the feature’s marketing page is something anyone could actually go and check.
Those are five separate engineering decisions, and each one is made badly by default if it is not made on purpose. This article works through each in turn, grounded in how a chip vendor, a mobile ML framework, or a shipping product actually documents the mechanism, rather than in how it tends to be summarised in a conference talk about a demo.
Matching compression to the silicon, not the model
The instinct is to pick one compression recipe — four-bit weights, say, or a favourite distillation pipeline — and apply it everywhere. But the target accelerator, not the model, decides what a compression strategy is even allowed to look like, because inference accelerators differ in which numeric formats they will actually execute at accelerated speed rather than falling back to a slow default path.
On Arm’s Ethos-U class microNPU this is not a preference but a hard constraint. Arm’s own guidance for optimising a model for the Ethos-U microNPU states plainly that the accelerator performs its calculations in 8-bit integer precision, and that a model must be quantised and then compiled with the Vela compiler before it will run on the NPU at all; operators that cannot be quantised or are otherwise unsupported fall back to running on the host CPU instead of the accelerator [4]. A model built around four-bit weights simply has no path onto that class of chip — the constraint is upstream of any accuracy argument.
Google’s LiteRT documentation is explicit that different post-training quantization techniques target different hardware. Dynamic-range quantisation statically converts weights to 8-bit integers while leaving activations in floating point, which the documentation reports as roughly four times smaller and two to three times faster, but on CPU only. Full-integer quantisation, which requires a representative calibration dataset, is what enables compatibility with integer-only accelerators such as the Coral Edge TPU and with microcontroller-class hardware. Float16 quantisation instead halves model size and lets a GPU delegate operate directly on the reduced-precision weights, but the documentation is explicit that it “does not reduce latency as much as a quantization to fixed point math” [3]. Three techniques, three different hardware targets, and no single one of them is simply “the” answer.
Apple’s Core ML Tools documentation draws the same line for its own silicon. It supports weight quantisation to 8-bit and 4-bit precision and activation quantisation to 8-bit, and it states specifically that “on newer hardware with A17 Pro or M4 chips… quantizing both activations and weight to int8 can leverage optimized compute on the Neural Engine” [2] — a claim scoped to a named hardware generation, not to Apple silicon in general, which matters if the feature’s install base includes older devices without that Neural Engine generation. Qualcomm’s AI Engine Direct SDK, meanwhile, exposes not one but three separate execution targets on a single chip — the Kryo CPU, the Adreno GPU and the Hexagon NPU — reachable either by targeting the SDK directly or by delegating from TensorFlow Lite or the ONNX Runtime [1]. On a Snapdragon device, “the compression strategy” is therefore not one decision but a family of them, one per backend, and the app has to know which backend a given device will actually route the feature to.
The practical order of operations follows from this. Decide the compression strategy after drawing the fleet’s actual accelerator mix, not before. If a meaningful share of the target fleet’s silicon has no path below 8-bit integer execution, spending engineering time on aggressive sub-8-bit compression is solving a problem most of the fleet does not have; the return is in a clean, fully-supported int8 conversion validated operator-by-operator against that specific backend’s support list, because a single unsupported operator silently falling back to CPU can erase the entire latency benefit the accelerator was chosen for.
A local-agent architecture that fails toward the cloud, not toward silence
Almost every shipping on-device feature is hybrid: a small local model handles most requests, and something else picks up the ones it cannot. The interesting engineering is not “if” but exactly how that handoff is built, and the clearest publicly documented example of the pattern currently comes from Microsoft’s guidance for Windows AI features. Its own developer documentation lays out, in working code, a three-tier resilience pattern: try the on-device Windows AI API first on supported Copilot+ PC hardware; if that path is unavailable, fall back to Foundry Local, which runs an open-source model on any Windows hardware; and if that is also unavailable, fall back to Azure AI in the cloud, described as “always available” [10]. The documentation is candid that this is a deliberate pattern for resilience rather than an incidental detail, stating it gives “Copilot+ users the best experience while keeping the feature working on all hardware” [10].
That example is useful precisely because it exposes what a “local-agent-with-fallback” architecture actually costs, which is easy to gloss over in a design review. Escalation is additive, not a substitution: a request that ultimately needs the cloud still pays for the local attempt first. Write
The consequence is structural rather than a tuning detail:
The architecture also has to decide what “the model can’t cope” means before a single request arrives, because that judgement has to be made from a signal available at the moment of the request, not from the correctness of an answer that does not exist yet. Google’s documentation for Gemini Nano running inside Android’s AICore system service is a useful contrast case precisely because it does not describe a per-request escalation path at all: AICore is described as isolating each request, not storing “any record of the input data or the resulting outputs after processing,” and as having no direct internet access of its own, with model downloads and any network activity routed through a separate system component [11]. Read structurally, that is a capability split rather than a router: the platform commits a defined set of tasks to the on-device model permanently, rather than deciding case by case whether an individual request is hard enough to send elsewhere. A capability split trades away the average-case quality gains a fine-grained difficulty router could offer, in exchange for a boundary that is stable, easy to explain to a user, and does not require a live decision that could itself leak information about which requests were judged sensitive or difficult. Both are legitimate designs; which one is right depends on whether the feature’s escalation decision is itself something the product is prepared to disclose.
Testing across a fleet, not a reference device
The device on an engineer’s desk is not the fleet, and the gap between the two is where sustained-load and cold-start assumptions actually break. Android’s own compatibility programme formalises this gap rather than leaving it to intuition: each Android release publishes a performance class in its Compatibility Definition Document, verified against real devices by the Compatibility Test Suite, and the documentation is explicit that “developers can find the device’s performance class at runtime and provide upgraded experiences that take full advantage of the device’s capabilities” — with the Jetpack Core Performance library exposing that level as a queryable value at runtime rather than something inferred from a marketing spec sheet [7]. A feature that gates its most expensive on-device path behind a runtime performance-class check, rather than a static allow-list of model numbers, degrades in a way that tracks the device’s real measured capability rather than a snapshot that goes stale the moment a new SKU ships.
Querying performance class only helps if the test matrix behind it is honest about device diversity, which is where cloud device farms earn their keep. Firebase Test Lab’s own documentation describes exercising an app “on devices installed and running in a Google data center,” across “a wide range of Android and iOS devices,” specifically so a team can find “issues that only occur on specific devices and configurations” [8] — the exact failure mode a single reference phone on a desk cannot surface by construction, because it is only ever one configuration. For an on-device AI feature the configurations that matter are not just screen size and OS version but accelerator generation, available memory under real background load, and thermal mass, none of which are visible from a spec sheet alone.
Standardised benchmarks close the last gap, because in-house device farms tend to accumulate whatever hardware was easy to acquire rather than what is representative of the shipped fleet. MLCommons’ MLPerf Mobile benchmark exists to give the industry a shared, comparable measurement across real hardware rather than each vendor’s own selectively favourable numbers; its sixth major release added generative on-device workloads running models including “Llama 3.2 1B Instruct, Llama 3.2 3B Instruct, Llama 3.1 8B Instruct,” together with expanded support for current-generation mobile silicon [5]. A parallel effort, MLPerf Client, does the same job for personal-computer form factors, evaluating LLM inference across current desktop, laptop and workstation GPUs and NPUs from multiple vendors under one methodology [6]. Neither benchmark substitutes for testing a specific feature on a specific fleet, but both give a team an external check on whether its own device farm’s numbers are plausible, and a shared vocabulary for describing hardware tiers that does not have to be reinvented per product.
Versioning a model across a fleet you do not fully control
A model that ships inside the app binary is simple to reason about and slow to fix: every user is running exactly the version that shipped with their currently installed app build, and improving the model means going through an app-store release cycle again. Three documented mechanisms exist to break that coupling, and each trades control for maintenance burden differently.
Apple’s Core ML Model Deployment, presented at WWDC 2020, delivers models to an app independent of its release cycle: models are grouped into collections tied to a feature, a collection is delivered to devices atomically so a feature’s models stay in sync with one another, and Apple’s own session materials state plainly that “each device’s system decides when to download the model in the background” — which is precisely why the same documentation insists that “your app should always have a fallback plan when the deployed models aren’t available” [12]. That fallback requirement is not boilerplate caution; it is a direct consequence of the delivery mechanism. Because the device, not the developer, decides the download moment, an app’s install base is guaranteed to contain a mix of model versions at any given time, and a feature that assumes every user is on the newest model will fail silently for whoever has not yet been served the update.
Firebase’s custom-model hosting occupies a similar niche for Android, iOS and Flutter: the console documentation describes deploying a TensorFlow Lite model and later updating it with a new file, with devices downloading the new version through the SDK rather than through an app-store release [13]. It is also a useful cautionary example for a different reason: at the time of writing, Firebase’s own documentation carries a notice that this service is deprecated and scheduled to shut down on 15 June 2027, with a recommended migration path to Cloud Storage for hosted TensorFlow Lite models [13]. A team that adopts an OTA model-delivery mechanism it does not operate has taken on a dependency with its own lifecycle, separate from the model’s own lifecycle, and that dependency can be deprecated out from under a shipped feature with two years’ notice.
The third posture removes versioning from the app’s control entirely. AICore is described as managing “the distribution of Gemini Nano and handles future updates,” explicitly so that a developer does not need to “worry about downloading or updating large models over the network, nor impact on your app’s disk and runtime memory budget” [11]. This is close to zero maintenance burden, purchased at the cost of zero control over when a capability changes underneath a shipped feature — a model behaviour shift becomes an operating-system update rather than a release the product team scheduled.
None of the three is categorically correct. A bundled model is the right choice when behavioural stability matters more than agility. A self-operated OTA collection is right when the team needs to iterate the model faster than the app releases but is willing to own the delivery infrastructure’s own lifecycle. A platform-managed model is right when the maintenance burden of the first two is unacceptable and the product can tolerate not controlling exactly when capability changes. Choosing among them without naming the trade explicitly is how a team ends up owning infrastructure it did not mean to own, or losing control it assumed it still had.
Measuring a privacy claim well enough to say it honestly
“Runs on device” and “your data stays private” are routinely used as if they were the same sentence, and a hybrid architecture is exactly where they come apart: the moment a feature has any escalation path to the cloud, “processed on device” is true of some requests and false of others, and a blanket privacy claim on the feature’s marketing page stops being accurate the instant the first request escalates.
The honest version of the claim is checkable rather than asserted, and two real production architectures show what a checkable version looks like. Apple’s Private Cloud Compute documentation is unusually specific for a privacy claim: it commits to publishing “the software images of every production build of PCC” for external inspection, states measurements are recorded in an “append-only and cryptographically tamper-proof transparency log,” and separately asserts that “PCC nodes cannot enable Developer Mode and do not include the tools needed by debugging workflows” [9]. Each of those is a falsifiable, specific claim about a mechanism — a published binary either matches what is running or it does not, a log either is append-only or it is not — rather than a general assurance that data is handled responsibly. Android’s AICore documentation makes a narrower but similarly specific claim for its own on-device path: AICore “does not have direct internet access,” and its data-handling design follows Android’s Private Compute Core principles [11].
The practitioner-level version of this discipline does not require building a transparency log; it requires being able to answer, for the specific feature being shipped, exactly which requests stay local, which escalate, and what crosses the wire when they do. That is a claim a network capture can verify directly: instrument the feature’s escalation path with a network tap or equivalent packet capture between the device and any endpoint it can reach, and confirm that a request classified as “local-only” produces no outbound traffic, and that a request that does escalate carries only the payload the privacy documentation says it carries — not device identifiers, not conversation history beyond what was disclosed, not telemetry bundled in alongside the model input. A privacy statement that has been checked this way, even informally, can say something narrower and truer than “your data stays private”: it can say which requests stay local, what leaves the device when one does not, and how a reader could verify that themselves. The narrower claim is more defensible than the broad one precisely because it is falsifiable, and a claim nobody can falsify is not evidence of anything, however reassuring it sounds.
Predictions, with the observations that would falsify them
These are forecasts, separated from the sourced analysis above. Horizon: 15 August 2028.
One. Mobile ML framework documentation will increasingly present quantisation choices as a hardware-compatibility table rather than an accuracy-versus-size trade-off alone, because the binding constraint most teams hit first is which backend a given format even runs on. Disconfirmed if major framework docs in 2028 still present precision choice primarily as an accuracy/size curve with hardware compatibility as a footnote.
Two. Three-tier local-then-cloud fallback patterns, of the kind now documented for Windows AI, will become the default reference architecture taught in official platform documentation for hybrid features generally, not just for one vendor’s stack. Disconfirmed if leading platform vendors in 2028 still document only a two-way local-or-cloud choice with no intermediate tier.
Three. Standardised device-fleet benchmarks like MLPerf Mobile and MLPerf Client will be cited in vendor privacy and performance disclosures as a way to substantiate fleet-wide claims, rather than remaining an internal engineering reference. Disconfirmed if consumer-facing product disclosures in 2028 still cite only single-device or best-case performance figures.
Four. Verifiable, checkable privacy claims — published binaries, transparency logs, or independently reproducible network-capture tests — will spread from frontier hybrid-AI vendors to mainstream mobile app privacy disclosures generally. Disconfirmed if mainstream app privacy documentation in 2028 remains dominated by unfalsifiable assurance language with no checkable mechanism behind it.
What to take away
None of the five decisions in this article is optional; each one gets made whether or not a team makes it on purpose. Compression made before the target fleet’s silicon is known is a guess dressed as an engineering decision. A fallback path that has never been forced to trigger has never actually been tested. A device farm that happens to contain whatever hardware was easy to acquire is not a fleet test. A model-update mechanism adopted without naming which of bundled, self-operated or platform-managed it is has an undeclared maintenance owner. And a privacy claim that cannot be checked on a wire is not a stronger claim than a checkable one — it is a weaker one wearing more confident language.
Ask, for any on-device feature about to ship: which accelerator generation does the compression strategy actually target, what happens on the request that the local model cannot answer, which real devices in the fleet were the fallback and thermal assumptions tested against, who decided when this user’s model version last changed, and what would a packet capture on this feature actually show. A feature that cannot answer those five is not finished shipping — it has only finished the demo.
