The Real Economics of Shipping a Model to a Device

The bill nobody itemizes

“Once you purchase the device, running an extra inference event is effectively free. In the cloud, the resources are extremely expensive.” Qualcomm’s chief financial officer, Akash Palkhiwala, made that argument publicly in December 2025, and as a statement about marginal cost it is correct ^[8]. It is also incomplete in a way that matters enormously to anyone actually deciding where a model should run, because marginal cost is one line in a much longer ledger, and Qualcomm has a direct commercial interest in the reader stopping at that line — it sells the silicon that on-device inference runs on, so a framing that makes edge compute look categorically cheap is also a framing that sells chips. That is not a reason to dismiss the claim; it is a reason to attribute it as a claim and keep reading the rest of the invoice.

The rest of the invoice has four line items that rarely appear in the same spreadsheet: the engineering cost of building and validating a matrix of model variants across device tiers, the compute cost of the distillation pipeline that produced the small model in the first place, the marginal cost difference between serving on-device and serving from the cloud, and the cost of getting updates out to a fleet that is already in customers’ pockets. Public discussion of “on-device versus cloud” almost always collapses these into the single marginal-cost comparison, because marginal cost is the easiest number to state and the most flattering to whichever side of the argument someone is arguing. This article itemizes the other three, attaches documented figures to each, and then puts all four into one break-even model.

Two decisions get conflated in casual accounts of this trade and need to be separated from the start. The first is whether compressing a large model into a small one is worth doing at all, independent of where the result gets served. The second is whether, having paid for that compression, the resulting small model should run on the device or continue to run in the cloud. Each has its own break-even condition, its own cost structure, and — as the following sections show with real disclosed figures — very different orders of magnitude.

The variant matrix: what “on-device” actually multiplies

A single cloud-served model is one artifact behind one API. A model shipped to devices is, in practice, several artifacts behind several build targets, and the count matters because every one of them has to be produced, evaluated and kept working.

Meta’s own account of shipping quantized Llama 3.2 models is a clean, disclosed example of the matrix rather than a hypothetical one. The company released two parameter sizes — 1B and 3B — each with two distinct quantization schemes, QLoRA-based quantization-aware training and SpinQuant post-training quantization, targeted specifically at Qualcomm and MediaTek systems-on-chip ^[12]. That is four separately built, separately shipped artifacts from a single starting checkpoint before a second chip vendor, a second operating system, or a second parameter size is even considered. Each of those four has to clear its own evaluation pass, because a quantization scheme is not guaranteed to degrade every task, every language and every prompt distribution identically — the whole reason two schemes were released side by side rather than one being simply designated as “the” quantized Llama is that they trade differently against each other, which is itself evidence that neither can be validated once and assumed to cover the other’s behavior.

Multiply that structure across the device tiers a consumer product actually ships to — flagship, midrange and a budget tier with a materially smaller neural accelerator and less memory bandwidth — and the number of build-times-tier combinations that need their own pass/fail sign-off before a release can go out stops being a rounding error in the schedule. This is the cost that a marginal-cost argument never counts, because it is paid once per release cycle regardless of how many queries any individual device ever serves.

The research literature on efficient deployment treats this multiplication as the central problem, not a footnote. Cai and colleagues frame the conventional alternative — designing or searching a specialized network from scratch for every hardware target — as “computationally prohibitive,” citing carbon emissions from that practice on the order of five cars’ lifetime driving, and propose training a single nested network from which specialized sub-networks for different devices can be extracted without additional training, a method they report reduces the required GPU-hours and associated emissions by “many orders of magnitude” relative to per-target search ^[4]. Read as an economics claim rather than a systems-research claim, that paper is a direct, quantified statement about the engineering cost of a variant matrix: not maintaining one is expensive enough to motivate an entire line of research into avoiding it.

Google’s MatFormer work generalizes the same idea to language models. By training nested feed-forward blocks of different widths inside one parent model, the method allows what the authors describe as hundreds of accurate smaller models to be extracted from a single trained parent, each validated as consistent with the family rather than as an unrelated one-off ^[6]. The economic content of that result is that the fixed cost of populating a device-tier matrix does not have to scale linearly with the number of tiers — but only for a team that architects around it from the start. Most teams that ship a small model today are still closer to Meta’s four-artifact structure than to a single elastic parent, which means the variant-matrix cost this section describes is the default, not the exception, across the industry as it currently operates.

Microsoft states the commercial logic for maintaining a family of sizes explicitly rather than treating it as an unfortunate byproduct of deployment constraints. Introducing the Phi-3 family, the company frames small models as sitting on a “quality-cost curve,” useful specifically for “resource constrained environments including on-device and offline inference scenarios” and for “cost constrained use cases,” alongside larger siblings aimed at different points on that same curve ^[10]. That is a vendor’s own account of why the matrix exists at all: different price points require different models, on purpose, which means the variant-maintenance cost is not a tax on a single “real” model but the price of a genuine product-line decision.

Two decisions, not one: distill, then place

With the variant-matrix cost established as real and ongoing, the first break-even question can be stated cleanly: is it worth training a smaller model against a larger teacher’s outputs at all, before any question of where the result will be served?

Disclosed training-run costs for this kind of compression are strikingly small in absolute terms. Stanford’s Alpaca project reported generating fifty-two thousand instruction-following examples from a larger model’s outputs for under five hundred dollars in API costs, then fine-tuning a seven-billion-parameter model on that data for under one hundred dollars of compute — a combined total the researchers themselves describe as under six hundred dollars ^[1]. Two years later, the NovaSky team at UC Berkeley trained a thirty-two-billion-parameter reasoning model, Sky-T1, competitive with an early OpenAI o1-preview on several benchmarks, for a stated cost under four hundred fifty dollars: nineteen hours on eight H100 GPUs ^[2].

Set those figures against what a frontier teacher model costs to produce in the first place. DeepSeek’s own technical report for DeepSeek-V3 states a training cost of $5.576 million, computed from 2.788 million H800 GPU-hours at an assumed rental price of two dollars per GPU-hour — and the report is explicit that this figure “include[s] only the official training of DeepSeek-V3, excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data” ^[5]. OpenAI’s Sam Altman put a similar order of magnitude on GPT-4, telling an audience that the model’s training cost “more than $100 million,” as reported by Fortune ^[11]. Epoch AI’s analysis of dollar training costs across machine learning systems published between 2009 and 2022 found costs growing by roughly half an order of magnitude per year for the field as a whole, with individual frontier systems in the one- to three-million-dollar range even years before DeepSeek-V3 and GPT-4 ^[3]. Teacher-class training cost is on a rising curve measured in millions to hundreds of millions of dollars; disclosed distillation and fine-tuning runs sit three to six orders of magnitude below that, measured in hundreds of dollars.

That gap explains why distillation investment almost never fails its own break-even test once a product actually ships. Write the total cost of training and then serving a distilled student model as

against the cost of simply continuing to serve the teacher directly for the same query volume,

Here $C_{\mathrm{distill}}$ is the one-time training cost, $Q$ is the number of queries served over the deployment’s life, and $c_{\mathrm{student}}$ and $c_{\mathrm{teacher}}$ are the marginal cost per query of serving each model, wherever that serving happens. The break-even query volume is

As a worked illustration only, not a claim about any real company’s ledger: take Alpaca’s disclosed 600 and assume, as an order-of-magnitude figure consistent with how frontier and compact API tiers are typically priced relative to each other, that a teacher-class model costs $0.002 per query to serve and a well-distilled student costs $0.0002 — a tenfold gap. Then $Q^{*}_{\mathrm{distill}} = 600 / 0.0018 \approx 333{,}000:KaTeX parse error: Can't use function '$1#039; in math mode at position 178: …ss model costs $̲0.002 per query… queries. Any production feature serving that many requests clears the investment in days, sometimes hours. This is the arithmetic reason distillation has become the default route to a deployable small model rather than a research curiosity: the pipeline cost is a rounding error against even modest cloud query volumes. Schwartz and colleagues’ “Green AI” argument — that the field should routinely report the “financial cost or ‘price tag’” of developing and running a model as a first-class evaluation criterion, not just its accuracy ^[7] — is worth restating here precisely because the headline distillation figures that circulate publicly are, by design, the cheapest line item in this whole accounting. Six hundred dollars is a true and verifiable number. It is also not the number that determines whether shipping the result to a device saves anyone money.

The break-even model: when the device actually wins

The second decision is the one this article’s assignment is really about, and it is a different comparison from the first. Given a small model that already exists — the distillation question already settled — should it run on the device or continue to run in the cloud?

Define $C_{\mathrm{dev}}$ as the fixed, per-release engineering and QA cost of producing and validating the device-tier variant matrix described above — the harness time, the pass/fail sign-off per tier, per quantization scheme, per chip target. Define $U$ as the cost of distributing model updates to the installed fleet, expanded in the next section. Define $c_{d}$ as the marginal cost of one additional on-device inference and $c_{\mathrm{cloud}}$ as the marginal cost of one additional cloud-served inference of the same model. Total cost over $Q$ queries under each placement is

and the break-even query volume — the point at which on-device placement stops costing more than cloud placement — is

This is the same fixed-cost-against-marginal-cost structure that governs any make-versus-buy or capital-versus-operating-expense decision, and it is the structure Qualcomm’s marginal-cost claim silently assumes has already been paid off. Palkhiwala’s statement that an extra inference event is “effectively free” once the device is purchased is a claim about $c_{d}$ alone; it says nothing about $C_{\mathrm{dev}}$ or $U$, both of which are paid by the model’s operator, not amortized into the price the customer already paid for the phone. Qualcomm’s own public argument for why $c_{\mathrm{cloud}}$ matters so much elsewhere in this equation is worth pairing with the CFO’s quote: the company’s 2023 hybrid-AI position paper states that “generative AI-based search cost per query is estimated to increase by 10 times compared to traditional search methods” ^[9] — a claim, again, from a party that benefits from cloud inference looking expensive, but directionally consistent with why $c_{\mathrm{cloud}} - c_{d}$ is large enough for the equation above to ever favor the device at all.

The equation also explains something the variant-matrix section set up: $C_{\mathrm{dev}}$ rises with every additional tier, chip target and quantization scheme a release has to validate, which means $Q^{*}_{\mathrm{device}}$ rises too. A feature built once and validated across three device tiers has a lower bar to clear than the same feature validated across six, holding query volume constant. This is a real, and rarely stated, reason product teams should expect on-device placement to make sense for a small number of high-frequency, broadly used capabilities — the ones that clear $Q^{*}_{\mathrm{device}}$ by a wide margin — and to make less sense for a long tail of niche features that a smaller fraction of the fleet ever triggers, even when each individual inference is nominally free at the margin. The fleet still has to receive the build whether or not most of its members ever use it, and that is exactly what $U$, the update-distribution term, prices.

What an update costs after the model has already shipped

$U$ is the least discussed term in this accounting and, at fleet scale, sometimes the largest. It can be approximated as

the number of devices in the fleet, times the bytes pushed to each device per update, times the price per byte of moving that data. Amazon’s published CloudFront pricing states data transfer out to the internet at 0.085 per gigabyte for the first nine terabytes per month in the United States, Mexico and Canada tier, after a free first terabyte [@aws-cloudfront-pricing]. As an illustrative calculation only, using that public list price rather than any negotiated enterprise rate: a modest 200-megabyte delta update pushed to a fifty-million-device fleet is ten million gigabytes of egress, or roughly $850,000 at that listed rate — before any volume discount, delta-patch efficiency gain, or CDN caching a real operator would negotiate. A full model re-download rather than a delta patch, at two gigabytes per device across the same fleet, would be an order of magnitude higher. Real operators pay less than this ceiling through bulk contracts and patch efficiency, but the ceiling illustrates why $U:KaTeX parse error: Can't use function '$1#039; in math mode at position 391: …ss, or roughly $̲850,000 at that… belongs in the equation at all: it is a cost that scales with fleet size regardless of how many of those devices ever actually invoke the feature being updated, which is precisely the same property that makes $C_{\mathrm{dev}}$ dangerous to ignore.

This is also the economic logic behind platform-level, rather than per-app, on-device models. Apple’s Foundation Models framework gives third-party developers access to the same on-device large language model that powers Apple Intelligence, and Apple’s own announcement states plainly that developers can build with it while “using AI inference that is free of cost” ^[13]. Read against the equation above, that phrase is not a claim that inference has no cost — it is a claim about who pays $C_{\mathrm{dev}}$ and $U$. Apple absorbs the variant-matrix engineering and the update-distribution bill once, centrally, at the operating-system level, and amortizes both across every app on every device rather than letting each app vendor pay its own share of the same fixed costs redundantly. A platform holder that controls the device, the operating system and the update channel can collapse a fleet-times-app multiplication down to a fleet-times-one, which is a genuine structural advantage unavailable to a single app vendor shipping its own model independently — and it is a different argument from the privacy or latency case for on-device inference; it is specifically about who is positioned to pay the fixed costs this article has been itemizing.

What companies say when they have to justify the number

Assembling the documented cases together, and characterizing rather than resolving the disagreement between them, is more useful than picking a winner. Qualcomm’s public position, voiced by its CFO and in its own hybrid-AI position paper, emphasizes marginal cost and a rising cloud-serving price as the reasons inference should move toward the device ^{[8, 9]} — a framing that is true as far as it goes and is also stated by a company whose chip revenue depends on that conclusion being drawn. Microsoft’s framing of the Phi family is different in kind: not “the device is nearly free,” but “different price points require different models,” a quality-cost curve spanning latency-bound, cost-constrained and capability-bound use cases simultaneously, most of which are not exclusively about on-device placement at all ^[10]. Apple’s framing is different again — a platform argument about who absorbs fixed cost, expressed to developers as inference being free of charge to them specifically because Apple, not the developer, has already paid $C_{\mathrm{dev}}$ and $U$ once at the OS level ^[13].

None of these three accounts contradicts the others; they describe different positions in the same equation from different vantage points, and the disagreement worth naming is about which term each party has the strongest incentive to foreground. A chip vendor foregrounds $c_{d}$ against $c_{\mathrm{cloud}}$. A model vendor selling API access to multiple sizes foregrounds the existence of a genuine quality-cost curve rather than a single right answer. A platform vendor foregrounds who pays $C_{\mathrm{dev}}$ and $U$ rather than whether they are paid at all. DeepSeek’s own disclosure practice is a useful corrective to take from all three: a company reporting a training cost explicitly flagged as excluding “prior research and ablation experiments” ^[5] is, in effect, warning its own readers that a headline dollar figure — theirs included — is very rarely the full engineering bill. That warning applies with at least as much force to marginal-cost claims about on-device inference as it does to training-cost claims about frontier pretraining.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: August 2029. They assume no regulatory mandate forcing on-device processing and no discontinuity in cloud GPU pricing.

One. Companies disclosing on-device deployment decisions will increasingly report a fully loaded cost — training plus variant-matrix QA plus update distribution — rather than a training-run figure or a marginal-cost claim alone, as procurement teams push back on comparisons that omit fixed costs. Indicator: vendor engineering blogs pairing a training-cost disclosure with a stated per-device or per-update cost. Disconfirmed if leading vendors in 2029 still publish only training-run costs or only marginal-cost claims without acknowledging the fixed side of the ledger.

Two. Elastic, nested-model training approaches in the style of Once-for-All and MatFormer will become the default route to a device-tier matrix for teams shipping more than two model variants, because the fixed engineering cost this article describes creates a direct financial incentive to adopt them. Indicator: major on-device model families documenting a single trained parent with multiple extracted sizes rather than independently trained variants. Disconfirmed if the leading on-device model families of 2029 are still produced as separately trained artifacts per size and per quantization scheme.

Three. Delta-patch and differential-update tooling for on-device model weights will become a named, budgeted line item in cloud infrastructure spend, distinct from general app-update bandwidth, as fleets running locally updated models grow. Indicator: CDN or cloud providers publishing pricing or tooling specifically for ML weight distribution. Disconfirmed if by 2029 model-weight updates are still bundled into undifferentiated app-update bandwidth with no dedicated tooling or pricing.

What to take away

The marginal-cost argument for on-device inference is not wrong; it is one term in a four-term equation, and the other three — variant-matrix engineering, distillation-versus-teacher economics, and update-distribution bandwidth — are each individually documented at real, checkable magnitudes in this article. Distillation itself is nearly always worth doing: disclosed training runs in the hundreds of dollars clear their own break-even point against even modest cloud query volumes almost immediately. Placing the result on a device is a separate, harder calculation, one that rises with every additional tier a release has to validate and every device already in the field that has to receive the update whether or not it ever calls the feature. Ask which term in $Q^{*}_{\mathrm{device}} = (C_{\mathrm{dev}} + U) / (c_{\mathrm{cloud}} - c_{d})$ a given vendor’s public statement is actually about, and whose incentive is served by leaving the other three out of the sentence.