How a Model Actually Gets Small Enough to Run on a Phone

A checkpoint is not shrunk, it is rebuilt three times

A companion piece in this series worked through why a phone constrains a model physically: a fixed memory budget, a bandwidth ceiling that batching cannot amortise the way a server’s can, a thermal wall that arrives in seconds rather than minutes, and a battery that prices every byte moved. None of that explains how an engineer actually gets from a checkpoint that only makes sense inside that envelope to one that fits. “Compression” is the word used loosely for the whole business, and it is misleading in a specific way: it suggests one operation, turned up or down, when a real pipeline is three mechanically distinct operations, each targeting a different kind of redundancy, applied in a specific order because each one changes the inputs the next one needs.

Knowledge distillation changes what the network is trained to predict. Pruning changes which parameters exist at all. Quantization changes the numeric grid those surviving parameters are allowed to land on. A model that ships on a phone has usually been through some version of all three, and the accuracy it ships with is the sum of three separate, partially-recoverable losses, each recovered by a different mechanism: a richer training signal, a retraining or reconstruction step, and either a calibration pass or a further round of training. This article works through each operation’s actual arithmetic, then through the order a practitioner runs them in, and why that order is not arbitrary.

The teacher, the student, and the loss that connects them

Training a small network from scratch on labelled data is a thin signal: for a given input, the target is a single correct class, everything else is simply wrong, and the loss treats all of the wrong answers as equally wrong. Hinton, Vinyals and Dean’s original distillation paper starts from the observation that a trained network’s mistakes are not equally distributed, and that this distribution is itself informative: a large, accurate model asked to classify an image of a bus assigns most of its remaining probability mass to “truck” and almost none to “carrot,” even when the image is unambiguously a bus ^[1]. That relative structure among the wrong answers is what the paper calls the network’s dark knowledge, and it is thrown away by training against hard labels alone.

Distillation recovers it by training the small “student” network against the large “teacher” network’s full output distribution rather than against the label alone. Write the teacher’s and student’s pre-softmax outputs for a given input as $z_t$ and $z_s$. A softmax with a temperature $T$ is

At $T=1$ this is the ordinary softmax used for prediction. Raising $T$ flattens the distribution, pulling the small probabilities assigned to wrong classes up toward visibility, which is exactly the dark knowledge the method wants to expose. The training objective blends two terms: ordinary cross-entropy against the true label, and a match between the student’s and teacher’s temperature-softened distributions, measured by KL divergence:

The $T^2$ factor is not decorative. Hinton and colleagues note that because the magnitude of the gradients produced by the soft-target term scales as $1/T^2$, multiplying the term by $T^2$ keeps the relative contribution of the two loss terms roughly stable as $T$ is tuned as a hyperparameter ^[1]. Without it, raising the temperature to expose more dark knowledge would simultaneously and silently shrink the gradient signal doing the exposing.

What this buys, mechanically, is a denser training signal per example. A hard label carries $\log_2 C$ bits at most, for $C$ classes, and in practice far less once you account for how skewed real label distributions are. A full teacher distribution over the same classes carries a graded judgment about every one of them, on every training example, for free. That is why a student trained this way can match a network many times its size on the same data budget: it is not learning faster, it is being taught with a richer curriculum. The modern survey literature on this technique describes it as the backbone of an entire ecosystem for building small open models from large proprietary ones, with the harder problem no longer the loss function above but generating enough varied prompts and teacher completions to distill against ^[14].

Unstructured pruning: removing the smallest weight, wherever it is

Distillation decides how the student’s weights are trained. Pruning decides which weights exist afterward. The oldest and simplest criterion is magnitude: rank every weight by $|w_{ij}|$ and remove the smallest fraction, on the reasoning that a weight close to zero contributes almost nothing to the layer’s output no matter what it is multiplied against. Applied once, magnitude pruning at any aggressive ratio breaks the network; applied iteratively, alternating a pruning step with a retraining step that lets the surviving weights absorb what the removed ones were doing, it can remove a large majority of parameters with only a small accuracy cost.

Frankle and Carbin’s lottery ticket hypothesis is the sharpest statement of why this iterative process works as well as it does. Starting from a fixed random initialisation, they show that a dense network contains sparse subnetworks that, “when trained in isolation, reach test accuracy comparable to the original network in a similar number of iterations,” provided those subnetworks keep their original initial weights rather than being reinitialised randomly ^[3]. Their winning tickets, found by iterative magnitude pruning, were routinely under a fifth of the original network’s size. The finding reframes what pruning is doing: not damaging a network and living with the damage, but searching for a substructure that was trainable to begin with and discarding the connections that were not carrying their share of the work.

This is unstructured pruning in the strict sense: the removed weights can be anywhere in a weight matrix, in no particular pattern. That irregularity is also its cost. A weight matrix with sixty percent of its entries zeroed, scattered arbitrarily, still occupies the same rectangular block of memory and the same dense matrix-multiply hardware unless something downstream is specifically built to skip zeros — sparse storage formats, sparse matrix kernels, or hardware with native support for irregular sparsity. Absent that support, an unstructured-pruned checkpoint is smaller to store and slower to reason about, but not necessarily faster to run.

Structured pruning: removing a whole unit at once

Structured pruning trades some of that compression ratio for a shape that ordinary dense hardware already knows how to exploit: instead of removing individual weights, it removes whole rows, channels, attention heads or layers, so that what remains is simply a smaller dense matrix, runnable by the same matrix-multiply kernel that ran the original.

Michel, Levy and Neubig’s study of transformer attention heads is the clean demonstration of where the redundancy for this actually lives. Testing what happens when heads are removed at inference time after training, they found that “a large percentage of attention heads can be removed… without significantly impacting performance,” with some layers reduced to a single head at negligible cost, and they trace the effect to how much a given head’s contribution is actually used downstream rather than to any property fixed at initialisation ^[4]. The finding generalises beyond attention heads: convolutional channels, feed-forward neurons and whole transformer layers all show the same pattern of graded, unevenly distributed importance, which is what makes structured removal viable rather than uniformly damaging.

The trade against unstructured pruning is direct. Removing a whole attention head at a fixed sparsity level typically costs more accuracy than removing the same fraction of individual weights chosen by magnitude, because a head-level decision is coarser than a weight-level one and cannot spare the genuinely useful weights inside an otherwise weak head. What it buys back is realised speed: a structured-pruned model’s FLOPs and memory traffic actually fall in proportion to what was removed, on hardware that requires no special support at all.

One-shot pruning without a retraining loop

Both of the classical routes above assume a retraining loop is affordable: prune, then run more gradient steps to let the network recover, repeat. For a network with tens or hundreds of billions of parameters, that loop is often not affordable, and a more recent line of work removes it by solving a smaller, local problem exactly instead of an approximate global one repeatedly.

SparseGPT poses pruning as a per-layer reconstruction problem. For a layer with weight matrix $W$ and a small calibration set of activations $X$, it looks for a sparse replacement $W'$ that keeps that layer’s output as close as possible to the original:

Rather than solving this by retraining, Frantar and Alistarh adapt a closed-form update derived from the layer’s second-order (Hessian) information, in the spirit of the older Optimal Brain Surgeon method, so that whenever a weight is removed the remaining weights in that row are analytically nudged to compensate for its absence. The result, reported for the GPT-family models tested, is that “large-scale generative pretrained transformer family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy,” with a 175-billion-parameter model prunable in a few hours on a single GPU, and the method extends to semi-structured 2:4 patterns that specific accelerator tensor cores can execute directly ^[5].

Wanda goes a step further and removes the reconstruction step too. Sun, Liu, Bair and Kolter’s pruning criterion ranks each weight not by its raw magnitude but by magnitude multiplied by the corresponding input activation’s typical size, “on a per-output basis,” on the observation that large-magnitude activation features are unusually concentrated in a small number of channels in trained language models, so a weight attached to a large, frequently-active input matters more than its raw size alone would suggest ^[6]. Because the criterion needs only a forward pass over a small calibration set to compute activation statistics, and no weight update at all, it is the cheapest of the three methods to run, at a small cost in accuracy relative to SparseGPT’s reconstruction.

Recovering the loss without full retraining

What these methods share, and what distinguishes them from the classical iterative loop, is where the recovery step happens. Magnitude pruning and the lottery-ticket procedure recover accuracy by running more gradient descent on labelled data after each pruning round — expensive, but general, and able to correct almost any kind of damage. SparseGPT recovers accuracy analytically, by solving a local least-squares problem that already knows what each remaining weight needs to become to compensate for its lost neighbours, using only a calibration set and no labels or backpropagation. Wanda recovers nothing at all; it simply chooses more carefully what to remove in the first place, trusting that the redundancy in a large trained network is generous enough that a good removal criterion needs no repair afterward.

This is the practical reason one-shot methods have displaced iterative retraining as the default for the largest models: retraining loop cost scales with the size of the network being retrained, while a calibration-only reconstruction’s cost scales with the size of the calibration set, which can be a few hundred examples regardless of how large the model is.

Quantization: rounding a checkpoint into integers

Distillation and pruning both operate on which numbers exist. Quantization operates on what those numbers are allowed to be. A trained checkpoint’s weights are ordinary floating-point numbers, typically 16 or 32 bits each, spread unevenly across a continuous range. Quantization replaces that continuous range with a small fixed grid of integers, and stores, alongside the integers, the arithmetic needed to convert back.

The standard affine mapping takes a real value $x$, a scale $s$ and a zero-point $z$, and produces an integer

where $\lfloor \cdot \rceil$ is round-to-nearest and $\hat{x}$ is the dequantised approximation actually used in arithmetic. Jacob and colleagues’ integer-arithmetic scheme, still the reference formulation for this mapping, showed that with weights and activations quantized this way, “inference can be carried out using integer-only arithmetic” end to end on ordinary integer hardware, with no floating-point unit required at all, which is what makes the technique valuable on the cheapest edge silicon rather than merely on the largest ^[7].

For weights, a common simplification is symmetric quantization around zero, with $z=0$ and the scale set directly by the largest magnitude present:

for a signed integer of $b$ bits. An 8-bit integer has $2^8=256$ representable levels; a 4-bit integer has $2^4=16$. That sixteen-fold reduction in available codes is the entire cost of quantization in one number. Treating the rounding error as approximately uniform over one quantization step $s$, its expected squared magnitude is the classical result

so halving the number of bits, which roughly doubles $s$ at fixed range, roughly quadruples the expected squared error per weight. That is why INT8 is usually described as close to free and INT4 is not: the error a network has to absorb does not grow gently as bits are removed, it grows quadratically in the step size, and every bit below eight is removed from an already-narrow budget.

Where the scale factor lives: per-tensor versus per-channel

The formula above needs one number, $\max(|x|)$, and the choice of what “one” applies to matters more than it looks. A single scale for an entire weight tensor is per-tensor quantization: cheap to store, one number per layer, but forced to size itself to whichever single row or output channel happens to have the largest values, wasting resolution on every other channel that has a smaller range.

Krishnamoorthi’s whitepaper on convolutional network quantization is the reference case for the fix: compute a separate scale for each output channel rather than one for the whole tensor, so that “per-channel quantization of weights and per-layer quantization of activations to 8-bits” lets each channel use the full resolution its own range actually needs, reporting classification accuracy for networks quantized this way that comes close to their unquantized floating-point originals, and finding that plain 8-bit weight quantization alone, without retraining, already delivers a fourfold reduction in stored model size ^[8]. The cost is bookkeeping: a per-channel scheme stores one scale per output channel instead of one per tensor, a small overhead against the savings from the bit width itself. As bit width falls, this stops being an optional refinement: at 4 bits, with only sixteen codes to spend, forcing every channel in a layer to share one scale can waste a meaningful fraction of those sixteen codes representing a range the channel never uses, and later post-training methods for language models build the per-channel or even smaller per-group granularity in as a default rather than an option.

Post-training quantization: calibrate once, round, ship

Post-training quantization, PTQ, takes a network that has already finished training, floating-point weights and all, runs a small unlabeled calibration set through it to observe the actual range of weights and activations, computes scales from that observation, and rounds. Nagel and colleagues’ white paper on the practice describes PTQ as “a lightweight push-button approach” that needs no retraining and no labelled data, typically sufficient to reach 8-bit precision with accuracy close to the floating-point original ^[9]. Google’s own LiteRT documentation, describing the runtime most on-device deployments actually use, states this in operational terms: full-integer post-training quantization uses “a representative dataset” to calibrate ranges and yields roughly a fourfold reduction in size with better than a threefold latency speedup on CPU, with a dynamic-range variant that quantizes weights only and needs no calibration set at all ^[15].

Below 8 bits, naive rounding starts to cost real accuracy, and three methods now define practice for getting language models lower without retraining. GPTQ solves the same kind of per-layer reconstruction problem SparseGPT uses for pruning, but for rounding: using approximate second-order information to decide, weight by weight, how to round so the layer’s output changes as little as possible, and Frantar and colleagues report quantizing a 175-billion-parameter model “in approximately four GPU hours” to 3 or 4 bits per weight with reported accuracy close to the unquantized original ^[10]. AWQ instead observes that “not all weights in an LLM are equally important,” identifies the roughly one percent of weight channels that matter most by looking at which channels see the largest activations, and rather than protecting them by leaving them at higher precision, rescales them before quantization so the rounding step itself does less damage to exactly those channels ^[11]. SmoothQuant addresses a different obstacle: large language model activations, unlike weights, contain outlier values that are hard to quantize accurately, so it migrates that difficulty from activations to weights with a mathematically equivalent rescaling before quantizing both, enabling accurate 8-bit weight and 8-bit activation quantization where naive per-tensor rounding of activations alone would fail ^[13]. Dettmers and colleagues’ LLM.int8() takes a related but distinct approach at inference time itself, using ordinary 8-bit vector-wise quantization for the great majority of values while carving out a small number of outlier feature dimensions to compute in 16-bit precision, reporting that “more than 99.9% of values are multiplied in 8-bit” while preserving full-precision-level accuracy on models up to 175 billion parameters ^[12].

Quantization-aware training: putting the rounding inside the loop

All of the methods above operate after training finishes, on a checkpoint that has never seen rounding during its own optimisation. Quantization-aware training, QAT, does the opposite: it simulates the rounding operation during training itself, inserting the quantize-and-dequantize step into the forward pass so the network’s own gradients see, and can adapt to, the exact numeric grid it will eventually be forced onto. Because rounding is not differentiable, this requires a workaround for backpropagation; the standard approach, part of the scheme Jacob and colleagues introduced alongside their integer-arithmetic inference format, treats the rounding step as the identity function during the backward pass, letting gradients flow through as if no rounding had happened, an approximation known as a straight-through estimator ^[7].

The trade against PTQ is cost against headroom. A calibration-only method needs no labelled data and no gradient steps at all; QAT needs the full training infrastructure and a further round of fine-tuning, often for as long as the original training run’s final phase. What it buys is accuracy at bit widths where PTQ starts to fail. Krishnamoorthi’s comparison found the accuracy gap against floating-point narrows to about one percent with QAT at 8 bits, and remains viable, at a real but bounded cost, down at 4 bits, where a purely post-hoc rounding step usually cannot fully recover ^[8]. In practice this makes QAT the fallback rather than the default: a team reaches for it specifically when a PTQ pass, tried first because it is nearly free, fails the accuracy bar at the bit width the deployment target actually needs.

The order an ML engineer actually runs this in

Put the three operations together and a specific, non-arbitrary order emerges, and it is close to the one Han, Mao and Dally established for convolutional networks a decade ago, when they showed that pruning, quantization and a final entropy coding step “work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy,” applied in that sequence, with a retraining pass after pruning and after quantization to “fine tune the remaining connections and the quantized centroids” ^[2]. Modern large-model pipelines follow an analogous shape, with one-shot methods replacing some of the retraining loops where the model is too large to afford them.

First, distillation, or the choice of a right-sized architecture trained against a teacher. This step happens first because it fixes the parameter budget everything downstream will work within, and because it is the most expensive step to redo: it requires running the teacher over a large training set and a genuine training run for the student. Everything after this step operates on the resulting dense checkpoint.

Second, pruning, with its recovery folded into whatever retraining budget is available. If a retraining loop is affordable, iterative magnitude pruning or a lottery-ticket-style search can be used, sparing structured units by importance the way head-pruning studies suggest ^{[4, 3]}. If it is not, a one-shot reconstruction method computes a sparse or semi-structured replacement directly from a calibration pass ^{[5, 6]}. Pruning happens before quantization because the quantizer’s calibration step needs to observe the actual weights and activations of the network that will ship, including whatever pruning has already removed; calibrating against a not-yet-pruned network would compute ranges for parameters that will not be there.

Third, quantization, tried cheaply first and escalated only if it fails. A practitioner runs post-training quantization first, because it is a calibration pass measured in minutes to hours rather than a training run measured in days, at whatever granularity, per-tensor, per-channel, or the even finer per-group schemes descended from it, the target bit width needs. If the resulting checkpoint clears the accuracy bar for the deployment target, typically true at 8 bits, the pipeline stops there. If it does not, typically the case pushing to 4 bits or below, the team escalates to quantization-aware fine-tuning, which in practice often reuses the same fine-tuning infrastructure and sometimes the same training job as the pruning-recovery step above it.

Fourth, packaging for the actual target. The quantized checkpoint, with its per-tensor or per-channel scale tables, is exported into whatever format the deployment runtime expects, verified against the specific accelerator, whether a mobile GPU, a dedicated NPU, or a microcontroller-class integer-only core, that will actually execute it.

Where practitioners disagree about the order

The order above is common, not universal, and the point of disagreement is genuine rather than cosmetic. One school runs distillation first because it believes a right-sized student, trained from the start against a rich teacher signal, converges to a solution that is already more compressible than an oversized network trained on hard labels and pruned down afterward; the distillation loss, on this view, is itself a regulariser toward redundancy the later stages can safely remove. A second school prefers to prune the teacher, or a large network trained conventionally, before any distillation happens, on the reasoning that starting from more capacity gives the pruning search more candidate substructures to choose from, and that a structured-pruned large model is itself a form of compression that does not require a separate, expensive distillation training run at all.

Neither position has settled the argument, and the honest description is that the choice depends on what is expensive in a given setting: teacher inference calls, if the teacher is a proprietary API rather than a network you hold weights for, or GPU-hours for a from-scratch pruning search, if the teacher is fully available. What is not in serious dispute is that quantization goes last: every one of the post-training methods above depends on calibrating against the actual weights and activations of the network that will ship, and that network is only fixed once distillation and pruning are done.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: 12 August 2028.

One. Pruning-recovery fine-tuning and quantization-aware training will increasingly merge into a single joint training stage rather than run as two sequential jobs, because both already require the same fine-tuning infrastructure and often the same data. Disconfirmed if major open compression toolkits and technical reports in 2028 still treat pruning-recovery and QAT as clearly separate sequential training jobs with no shared loop.

Two. Semi-structured N:M sparsity support will spread from a narrow set of datacentre accelerators to being commonly supported on edge NPUs, because it is close to the only sparsity pattern that yields a real speedup without a bespoke sparse kernel. Disconfirmed if 2028 edge NPU documentation still supports only dense INT8/INT4 execution or whole-channel structured pruning, with no N:M sparse execution path.

Three. Below 4 bits, post-training quantization will increasingly be reported as a baseline rather than the shipped method, with quantization-aware training or distillation-aware low-bit training becoming the default at 2 to 3 bit widths. Disconfirmed if a widely adopted 2- or 3-bit deployed language model in 2028 is produced by post-training quantization alone, with no quantization-aware retraining step.

Four. Calibration-set composition for post-training quantization will become a disclosed part of model documentation, because activation-aware methods have already shown that quantization quality depends on what is in that small sample, not only on how many bits are used. Disconfirmed if leading model cards in 2028 still ship quantized artifacts with no disclosure of what calibration data was used to compute their scales.

None of these requires a new compression technique to appear. They follow from how the three operations already interact.

What to take away

A small deployed model is not a large one with the volume turned down. It is a checkpoint that has been rebuilt three separate times by three separate mechanisms: trained against a richer teacher signal rather than bare labels, stripped of weights or whole structural units by a criterion that ranks their contribution and a procedure that compensates the survivors, and rounded onto an integer grid narrow enough that the rounding error itself had to be budgeted for by bit width, by granularity, and by whether the rounding happened before or during a further round of training. Each stage recovers its own loss by its own mechanism, an argument in Hinton’s temperature, a reconstruction in SparseGPT’s Hessian update, a calibration pass or a straight-through gradient in a quantizer, and the order those stages run in is not convention. It is a dependency graph: what a quantizer calibrates against has to already reflect what pruning removed, and what pruning removes has to already be trained into a network shaped by the teacher it learned from.

Ask, of any small model, which of the three operations produced it, in what order, and what each one’s loss was measured and recovered against. A claim that a model was simply “made smaller” has not yet answered any of those three questions.