A History of Small and On-Device AI

Before the phone could run anything, it had to fit

By 2012, the deep convolutional networks winning image-recognition contests were built to run on a machine that plugged into a wall. AlexNet held around sixty million parameters; the VGG networks that followed pushed past a hundred and thirty million. None of that was designed against a budget, because a research group with a rack of GPUs did not need one. A phone is a different machine: a few hundred megabytes it can spare for one running model, a battery that has to last the rest of the day, and no fan to carry away whatever heat the chip produces. The interesting history here is not “AI got better and eventually it fit on a phone.” It is five separately dated, separately motivated engineering answers — how to design a network’s structure economically, how to build a chip for it, how to run a detector continuously without ever asking a server, how to compress what a large model already learned, and, only very recently, how to make a full language model small enough for a phone still worth using — that ran in parallel for the better part of a decade before they arrived, together, in the same product.

Building small on purpose: SqueezeNet and MobileNet

The first attack on the problem did not compress an existing large network; it designed a new one whose entire point was never needing compression. Iandola and colleagues’ SqueezeNet, released in February 2016, reported matching “AlexNet-level accuracy on ImageNet with 50x fewer parameters,” and, once its weights were compressed further, a model “less than 0.5MB (510x smaller than AlexNet)” ^[2]. The architecture’s central move was the “fire module”: a squeeze layer of 1×1 convolutions that shrinks the number of input channels before a wider expand layer processes them, so the network spends its parameter budget on channel width only where that width is cheap. The authors were explicit that the motivation went beyond a leaderboard score: a smaller model needs “less communication across servers during distributed training,” less bandwidth “to export a new model from the cloud to an autonomous car,” and is more deployable on FPGAs and other hardware with limited memory ^[2] — a list of constraints that has almost nothing to do with accuracy and everything to do with where the model has to live.

MobileNet, published by Howard and colleagues at Google fourteen months later, generalized the same idea into a reusable building block rather than one bespoke network. A standard convolutional layer filters and combines its inputs in a single step; MobileNet’s depthwise separable convolution splits that into a depthwise layer that filters each input channel on its own, followed by a 1×1 pointwise convolution that recombines the results ^[1]. For a $D_K \times D_K$ kernel, $M$ input channels, $N$ output channels and a $D_F \times D_F$ feature map, a standard convolution costs $D_K \cdot D_K \cdot M \cdot N \cdot D_F \cdot D_F$ multiply-adds. The paper gives the depthwise separable replacement’s cost, and its ratio to the standard cost, directly:

With the near-universal $3\times3$ kernel, that ratio works out to roughly an eighth to a ninth of the original computation, for what the paper reports as a small accuracy cost ^[1]. MobileNet then added two further dials rather than shipping one fixed shape: a width multiplier $\alpha$ that uniformly thins the number of channels at every layer, and a resolution multiplier that shrinks the input image, letting a developer choose a point on an accuracy-versus-latency curve instead of accepting whatever a single published checkpoint offered ^[1]. That is the detail worth keeping from this stage of the history: SqueezeNet showed that a deliberately designed structure could match a much larger network’s accuracy at a fraction of the parameters; MobileNet turned the same insight into a convolution that could be dropped into any architecture, with knobs that let one trained design be retargeted across a whole range of phones rather than retrained for each.

Silicon catches up, all at once, in 2017

An efficient architecture still needs somewhere efficient to run. Through the early 2010s, a phone’s neural-network inference — what little of it existed — ran on the same general-purpose CPU and GPU used for everything else, competing with the rest of the operating system for the same execution units. That changed inside one calendar year, when three of the largest phone chip vendors each shipped dedicated on-device inference hardware within about eight months of one another.

Huawei went first, unveiling the Kirin 970 at IFA in Berlin on 2 September 2017 as, in the company’s own description, a system-on-chip built around a “dedicated Neural Processing Unit (NPU)” ^[10]. Huawei’s released figures claimed that, compared with a quad-core Cortex-A73 CPU cluster doing the same work, “the Kirin 970’s new heterogeneous computing architecture delivers up to 25x the performance with 50x greater efficiency,” illustrated with an image-recognition benchmark of “2,000 images per minute” ^[10] — a vendor benchmark, not an independently reproduced one, but a clear statement of what a dedicated block was being built to do.

Ten days later, Apple’s A11 Bionic, announced 12 September 2017 for the iPhone 8, iPhone 8 Plus and iPhone X, added what Apple called a Neural Engine: a dual-core block “purpose-built for machine learning” that Apple reported performs “up to 600 billion operations per second for real-time processing,” used at launch to run the networks behind Face ID and Animoji ^[8]. Qualcomm’s contribution arrived from a different direction — not a brand-new inference block, but a software layer opened onto compute that was already sitting on the chip. On 24 July 2017, Qualcomm announced general release of its Snapdragon Neural Processing Engine SDK, letting a developer target the Hexagon digital signal processor, the Adreno GPU, or the Kryo CPU inside a Snapdragon 835 from one framework, importing models trained in Caffe, Caffe2, or TensorFlow ^[9].

None of these three efforts is directly comparable to another on the numbers each company chose to publish, and none of the reported figures were produced by an independent benchmark; they are vendor claims about vendor hardware, offered here as exactly that, not as a ranking. What is comparable, and does not depend on trusting any one company’s marketing, is the timing: dedicated silicon for neural-network inference went from essentially absent on consumer phones to present on flagship chips from three separate vendors inside less than a year, arriving almost immediately behind the architectures described above. That sequencing is unlikely to be coincidence. A convolution redesigned to spend its multiply-adds on cheap operations is only worth building a chip around once there is a network shaped that way to run, and MobileNet and SqueezeNet gave chip design teams a specific, publishable workload to target right as those teams were already deciding where next year’s die area would go.

Listening without waking the network

A complete history has to include the case where on-device inference was never optional, because it is the case that forced the discipline the rest of this article assumes. A phone that must recognize a wake word — “Hey Siri,” “OK Google” — cannot send audio to a server continuously; that would mean streaming a live microphone feed to the network for as long as the phone is on, at a battery and privacy cost no vendor would accept and few users would tolerate. Keyword spotting had to be small, low-power and local from the start, years before “on-device AI” was a phrase anyone used commercially.

Chen, Parada and Heigold’s 2014 paper stated the requirement in its opening line: “a keyword spotting system with a small memory footprint, low computational cost, and high precision” ^[5]. Their approach trained a deep neural network to predict a keyword, or its constituent sub-word units, directly from acoustic features, replacing the sequence search of a Hidden Markov Model with a simpler posterior-handling step. The paper reports “45% relative improvement with respect to a competitive Hidden Markov Model-based system,” and “39% relative improvement” specifically under added babble noise ^[5]. The authors were explicit that the appeal was architectural as much as accuracy-driven: “Running the system on the device avoids latency and power implications with connecting to the server for recognition” ^[5] — the deployment argument for edge inference, stated plainly, years before it became a marketing category.

Apple’s account of the detector behind “Hey Siri,” published in 2017, shows what that constraint looks like once it is actually shipped at scale, across hundreds of millions of devices. Rather than run any neural network on the phone’s main application processor while listening continuously, the detector runs on the “Always On Processor (AOP) (a small, low-power auxiliary processor, that is, the embedded Motion Coprocessor),” which has its own independent access to the microphone signal ^[7]. Detection happens in two passes: a small first-pass network on the AOP screens continuously, and only if it fires does “the main processor” wake up to run “a larger DNN” that checks the result more carefully ^[7]. Apple describes networks of “five hidden layers, all the same size: 32, 128, or 192 units depending on the memory and power constraints” — the exact unit count chosen by how much power budget a given device generation can spare for continuous listening ^[7]. On Apple Watch, the constraint tightens further: a single-pass detector runs “only when the watch motion coprocessor detects a wrist raise gesture,” consuming “approximately 5% of the compute budget” ^[7].

Pete Warden’s Speech Commands dataset, released in 2018, gave the field a shared benchmark built specifically for this narrower problem, distinct from general-purpose speech recognition, explicitly meant to “help train and evaluate keyword spotting systems” ^[6]. Its arrival mattered for an ordinary but important reason: before a shared dataset existed, every keyword-spotting paper reported results on a private, product-specific corpus, and comparing one paper’s claimed accuracy against another’s was close to meaningless.

Compressing what a bigger model already knows

The technique now central to nearly every capable small model on the market did not originate in deep learning at all, and it is nine years older than the paper most people cite for it.

In 2006, Buciluă, Caruana and Niculescu-Mizil published “Model Compression” at the ACM SIGKDD conference, addressing a problem their abstract states in terms this article’s subject would recognize immediately: ensemble methods perform well but are too large and slow for use in applications “where storage space is at a premium (e.g. PDAs), and where computational power is limited (e.g. hearing aids)” ^[3]. Their method trained a single compact neural network to mimic the output of a large ensemble, using a technique they called MUNGE to generate enough synthetic training data for the small network to learn the ensemble’s decision surface, rather than only the labels available in the original, often much smaller, training set. Their reported result states the trade directly: “we are able to train neural nets that are a thousand times smaller and faster than ensemble selection ensembles, but which have nearly the same performance as the far more complex ensembles” ^[3]. That paper predates AlexNet by six years. The constraint driving it — a hearing aid, a PDA, a device that cannot hold or run a large model — is exactly the constraint this entire history is about, reached from ensembles of decision trees rather than neural networks, nearly a decade before the deep-learning era most people associate the idea with.

Hinton, Vinyals and Dean’s 2015 paper, “Distilling the Knowledge in a Neural Network,” is the one the field actually cites, and it earned that position by generalizing the 2006 idea and giving it a mechanism simple enough to fit into any neural network’s training pipeline. Instead of training a small model against a large model’s hard output labels, distillation trains it against the large “teacher” model’s full, softened probability distribution over classes — the softmax output computed at a raised temperature $T$:

where $z_i$ are the teacher’s pre-softmax logits. “Using a higher value for $T$ produces a softer probability distribution over classes” ^[4], and it is precisely that softness — the relative probabilities the teacher assigns to the wrong answers, not only the single correct label — that carries information a hard label discards: how confidently a teacher model preferred one wrong answer over another says far more about the shape of its decision boundary than one correct label does. The paper states the mechanism directly: “Knowledge is transferred to the distilled model by training it on a transfer set and using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax” ^[4]. On a speech-recognition acoustic model, the authors reported that “more than 80% of the improvement in frame classification accuracy achieved by using an ensemble of 10 models is transferred to the distilled model” ^[4] — most of what ten expensive models knew, recovered in a single model built to run where the ensemble could not.

A phone that can write: capable small language models arrive

The four threads above — an architecture that spends its parameter budget economically, silicon built to run it, a design discipline that treats offline and low-power operation as non-negotiable, and a technique for compressing a large model’s behaviour into a small one — are not historical background to the current wave of small on-device language models. They are its literal, documented ingredients.

Microsoft’s Phi-3 technical report titles itself “A Highly Capable Language Model Locally on Your Phone” and states that its smallest member, phi-3-mini, is “small enough to be deployed on a phone,” at 3.8 billion parameters trained on 3.3 trillion tokens, reporting “69% on MMLU and 8.38 on MT-bench” ^[11] — self-reported figures from the model’s own developer, offered here as a claim rather than an independently reproduced result, but a clear statement of the design target. Liu and colleagues’ MobileLLM, from Meta, made the architectural argument explicit for models below a billion parameters: rather than treat parameter count as the only lever, the paper finds that “model architecture” plays a pivotal role for sub-billion-scale language models, and that “deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms” produced accuracy gains of “2.7% and 4.3%” over the prior best 125-million- and 350-million-parameter models ^[12] — the same design conviction SqueezeNet and MobileNet argued for vision networks a decade earlier, applied to language models at a scale small enough to matter on a phone.

Google’s Gemma 3n, documented in a 2025 developer guide, makes the memory constraint itself the headline. Its MatFormer architecture is described as “a nested transformer built for elastic inference,” shipped in two sizes whose raw parameter counts — “5B raw parameters” for the smaller E2B, “8B” for the larger E4B — substantially overstate what actually has to sit in the accelerator’s constrained memory, because the model’s Per-Layer Embeddings can be “loaded and computed efficiently on the CPU,” leaving “only the core transformer weights (approximately 2B for E2B and 4B for E4B)” resident where memory is scarce, with the guide stating operation “with as little as 2GB (E2B) and 3GB (E4B) of memory” ^[13]. Apple’s own foundation-model report describes a comparable choice from the other direction: alongside a larger server-side model reserved for its Private Cloud Compute system, Apple built “a ~3 billion parameter model designed to run efficiently on devices,” sized specifically for the phones and tablets running Apple Intelligence rather than for a benchmark leaderboard ^[14].

None of these four reports is independently verified against the others, and none should be read as settling a ranking between vendors — they measure different benchmarks, disclose different amounts of method, and were written by the teams selling the systems they describe. What is verifiable without trusting any single one of them is the shared shape of the claim: a model whose designers state, as a primary design constraint, that it must run inside a phone’s memory and power budget rather than a server’s, using exactly the tools this history has already introduced — an architecture designed to be cheap rather than compressed after the fact, run on accelerator silicon descended from the 2017 generation, trained in significant part by distilling a larger model’s behaviour, under a design discipline about offline operation and battery cost that keyword spotting established as non-negotiable a decade earlier.

What converged, and what still has not

None of the five threads above were run as one project. Efficient-architecture research came out of computer-vision groups optimizing for ImageNet leaderboards and mobile deployment simultaneously. Dedicated inference silicon came out of chip teams responding to a workload that vision researchers had just made concrete. Keyword spotting came out of speech teams solving a narrower, older problem — a wake word — under a power constraint that had no cloud escape hatch. Distillation came out of an ensemble-learning paper from 2006 that never mentions a smartphone, aimed at PDAs and hearing aids that barely exist as product categories today. What makes this a single history rather than four unrelated ones is that every one of those threads was solving some version of the same equation — a fixed, non-negotiable envelope of memory, computation and power — a decade before there was a language model small and capable enough to need all four solutions at once.

It is worth being explicit about what remains a vendor claim rather than an independently settled fact in this account. Gemma 3n’s “effective” 2GB and 3GB footprints depend on Google’s own accounting of what counts as resident accelerator memory, not a third-party audit. Phi-3’s and MobileLLM’s benchmark numbers are self-reported by their own developers on evaluation suites those developers chose. The 25x and 50x figures Huawei published for the Kirin 970, and the 600-billion-operations-per-second figure Apple published for the A11’s Neural Engine, are each one company’s own benchmark of its own hardware. None of that means the claims are false; it means a reader comparing across them is comparing marketing disclosures of unequal depth, not a controlled study, and the honest response is to note that gap rather than resolve it by picking a winner among incomparable numbers.

Predictions, with the observations that would falsify them

These are forecasts, kept separate from the sourced history above. Horizon: 12 August 2028. Assumptions: no discontinuity in mobile memory technology reaching consumer devices, and no regulatory mandate forcing or banning local inference.

One. Technical reports for phone-scale language models will increasingly disclose which specific technique — architecture search, quantization-aware training, or distillation from a named teacher — produced most of a small model’s reported capability, because the current practice of listing all three without attribution is becoming harder to defend as vendors’ methods diverge. Indicator: reports with a component-level ablation crediting each technique separately. Disconfirmed if leading small-model reports in 2028 still describe their training recipe only in aggregate.

Two. Dedicated on-device NPUs will increasingly be reported in a metric comparable across vendors — most plausibly a measure of inferences per unit of energy — displacing today’s vendor-chosen operations-per-second figures, because the current figures are not comparable to one another and procurement teams have started to notice. Disconfirmed if the dominant published metric for mobile NPUs in 2028 is still a raw operations-per-second number chosen independently by each chipmaker with no efficiency term.

Three. Distilling a large proprietary model’s outputs into a small openly released one will become the subject of explicit terms in frontier providers’ usage policies, because the practice traced to the 2006 and 2015 papers above is now central to how competitors build capable small models. Disconfirmed if by 2028 major providers’ terms of service remain silent on distillation from their models’ outputs.

Four. The distinction this article draws between “designed small” — architectures like SqueezeNet and MobileNet built economical from the outset — and “compressed small” — distillation and quantization applied after training — will collapse in practice, with nearly every shipped small model in 2028 using both together rather than relying on either alone. Disconfirmed if a meaningful share of newly released small on-device models in 2028 still rely on only one of the two approaches.

What to take away

The phone-scale language models arriving now are not a sudden capability. They are the point at which five independently dated engineering answers — an economical convolution published in 2016 and 2017, dedicated chips built for that shape of computation within the same calendar year, an offline-first design discipline that proved itself on a wake word years earlier, and a compression technique first published in 2006 for hearing aids and pocket organizers — finally had a workload demanding enough to need every one of them at once. Read a claim about a new small model against that lineage rather than against the assumption that it appeared from nowhere: ask which of the four older techniques it is actually using, whose benchmark the reported number came from, and whether the constraint it is solving is, in the end, the same one a PDA or a hearing aid solved first.