Shrink It, Train It Small, or Search for It: The Main Strategies for Small Models, Compared

Four different problems wearing one name

“Make it small” names a target, not a method. In current practice, teams reach that target from four structurally different starting points, and the difference in starting point matters more than any single leaderboard number suggests. The common error is to weigh the outputs of all four — benchmark scores at roughly matched parameter count — as though they were four entrants in one competition. They are not. Each strategy holds a different resource fixed as “already paid for” and treats a different resource as the one being spent, and which resource is scarce for a given team determines which strategy is even applicable, before any question of which one is “better.”

Four strategies cover most of the field’s current, documented practice.

Compress an existing large model. Distillation, quantization, and pruning treat a large pretrained model’s already-completed training as an asset to copy from rather than a result to reproduce.

Train a small model on purpose. A data-efficient recipe commits the entire compute budget to a model that is small from the first optimizer step, spending on data curation rather than on parameters or tokens inherited from a bigger design.

Search architecture space directly. Neural architecture search treats the network’s structure — not only its size — as the object being optimized, against measured cost on the actual target hardware rather than an inherited design.

Route around density. Sparse mixture-of-experts decouples how much a model has learned from how much compute a single token costs, activating a slice of a much larger parameter store per request instead of shrinking the store itself.

The rest of this article works through each strategy’s own stated rationale, its documented cost tradeoffs, and its documented limitations, in the words and numbers of the papers that established it. It closes by making explicit why comparing them on one scale produces a misleading answer, and by stating what would have to be observed over the next two years to show that framing wrong.

Compress what already exists

The oldest of the four strategies starts from an asset that is already paid for: a large model has already been trained, at whatever cost that took, and the cost is sunk. Compression treats the sunk cost as something to copy rather than repeat. Hinton, Vinyals, and Dean set out the core argument in 2015, motivated by a practical deployment problem with large ensembles: “making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users” ^[1]. Their proposed fix — train a small “student” network to match a large “teacher” network’s full output distribution, not merely its top label — is usually written with a softened distribution and a temperature parameter:

where $z_t$ and $z_s$ are the teacher’s and student’s output logits, $\sigma$ a softmax, $T$ a temperature that softens the distribution, and $y$ the ground-truth label. The second term is the entire point of the method: it transfers the relative probability the teacher assigns to every wrong answer, not just which answer was right, a far richer training signal per example than a raw label alone provides. That is the rationale, stated plainly by the paper that introduced it.

The cost tradeoff is the reason this strategy exists at all, and it has since been quantified directly for language models. Xia, Gao, Zeng, and Chen pruned an existing pretrained LLaMA model structurally and then continued training it briefly, rather than training a same-sized model from a random initialization, and reported that the resulting Sheared-LLaMA models were competitive “while requiring only 3% of compute compared to training such models from scratch” ^[3]. Compression’s core promise is exactly that ratio: most of the expensive work — learning general structure from raw data — has already happened inside the large model, and copying it out is documented to be far cheaper than repeating it. Combining several compression techniques compounds the saving further. Han, Mao, and Dally’s Deep Compression paper, applying pruning, trained quantization, and Huffman coding together, reported a “storage requirement of neural networks by 35x to 49x without affecting their accuracy,” alongside “3x to 4x layerwise speedup and 3x to 7x better energy efficiency” across CPU, GPU, and mobile GPU platforms ^[2]. None of that required retraining a model from nothing.

The strategy carries two structural limitations that follow directly from how it works, not from any one paper’s shortcoming. First, it requires access to the large model itself, or at minimum to its outputs at sufficient fidelity to train against — a dependency none of the other three strategies share. If the frontier model is API-only, gated, or simply unavailable to the team doing the compressing, the achievable fidelity of “dark knowledge” transfer is bounded by whatever access is actually granted. Second, and more fundamentally, a compressed model is trained to match its teacher, not trained against the underlying task from scratch — whatever the teacher gets systematically wrong, declines to answer, or was never taught, the student is optimized to reproduce rather than to independently discover a better answer. This is a structural consequence of the objective in the equation above, not an empirical finding reported by any single paper cited here: the loss function rewards matching $z_t$, and has no term that would push the student past it.

Train small on purpose

The second strategy shares none of the first strategy’s dependency on an existing large model. It commits the full training budget, from the first optimizer step, to a model whose size is fixed at the deployment target rather than arrived at by shrinking something else.

Its intellectual foundation is the same scaling-law literature that shaped how large models are trained, applied in the opposite direction. Hoffmann and colleagues showed that contemporary large language models had been trained on too little data relative to their parameter count, and that for a fixed training budget, model size and training tokens should grow in roughly equal proportion — “for every doubling of model size the number of training tokens should also be doubled” ^[4]. The underlying loss law is commonly written as

with $N$ parameters and $D$ training tokens. Chinchilla’s question was: given a fixed compute budget $C$, what $N$ and $D$ minimize $L$? The data-efficient small-model strategy asks a different question with the same law: given a fixed, small $N$ set by the deployment target, what choice of $D$ — and, crucially, what quality of $D$ — minimizes $L$? Fixing the small side of the equation first and spending the freed budget on data rather than on parameters is the strategy’s entire premise.

Two documented recipes show how far that substitution can be pushed. Gunasekar and colleagues built phi-1, a 1.3-billion-parameter model, on “a selection of ‘textbook quality’ data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens),” trained in four days on eight accelerators, and reported it reached 50.6% pass@1 on HumanEval and 55.5% on MBPP ^[5] — figures the paper’s own authors report for their own model, stated here as their claim. Eldan and Li pushed the same lever toward its floor rather than toward its ceiling: by restricting the training corpus to stories using “words that a typical 3 to 4-year-old usually understands,” they found coherent, grammatical narrative generation emerging in models “below 10 million total parameters” ^[6]. Read together, these two results make the same point from opposite directions: there is no single answer to “how small can a capable model be,” because the achievable floor depends on how narrowly the target domain is drawn, and domain-narrowing is a lever this strategy controls directly, in a way the other three mostly do not.

The cost this strategy pays for that lever is documented candidly in phi-1’s own conclusion and appendix. The authors state plainly that “phi-1 is specialized in Python coding, which restricts its versatility compared to multi-language models,” that it “lacks the domain-specific knowledge of larger models such as programming with specific APIs or using less common packages,” and that because of “the structured nature of the datasets and the lack of diversity in terms of language and style, phi-1 is less robust to stylistic variations or errors in the prompt,” with performance dropping “significantly as the length of the prompt increases” ^[5]. That is the direct cost of the strategy’s central move: narrowing and curating the training distribution buys capability per parameter, and the price is fragility outside that distribution. TinyStories takes the same trade further and makes it the entire premise rather than a side effect, restricting the domain so tightly that “coherent” is achievable at a scale where general-purpose language ability plainly is not ^[6]. Set against compression, the tradeoff is symmetric: this strategy pays full training cost from the first token, with no discount for reusing a larger model’s already-learned structure, but in exchange it never inherits an architecture, a data mixture, or a set of blind spots chosen for a different, larger deployment.

Search the space of small architectures

The first two strategies both largely accept a given architecture — one inherited from the model being compressed, the other a conventional small transformer chosen by hand. The third strategy makes the architecture itself, not only its parameter count, the object being optimized, searching for structures that are efficient on the hardware the model will actually run on.

Zoph and Le established the modern form of the idea: a controller network, trained by reinforcement learning, proposes candidate child-network architectures, each of which is trained and evaluated, with the resulting performance used as a reward signal to improve the controller ^[7]. Formally, a NAS run of this kind is a constrained, nested optimization:

where $\mathcal{A}$ is a search space of candidate architectures designed in advance by the researchers, $g(a)$ some measured deployment cost of architecture $a$, and $B$ a budget the target device imposes. Every term in that equation is a documented design choice, and the choice of $g$ turns out to be where the edge-specific literature does its most important work.

The cost of the outer search, in Zoph and Le’s original formulation, is the strategy’s headline limitation, and it is stated in the paper’s own description of its training setup: the method trains “800 networks being trained on 800 GPUs concurrently at any time” ^[7]. That expenditure is on top of, not instead of, whatever it costs to train the architecture eventually chosen — an outlay far beyond what either compression or a data-efficient from-scratch recipe requires, and one that most teams cannot repeat for every new hardware target. Two later, edge-specific results respond to that limitation in different ways.

The first response changes what $g(a)$ measures. Tan and colleagues built MnasNet’s rationale explicitly around the observation that operation counts are a poor stand-in for real efficiency, so the search “directly measures real-world inference latency by executing the model on mobile phones,” and reported an architecture 1.8x faster than MobileNetV2 at 0.5% higher accuracy, and 2.3x faster than NASNet at 1.2% higher accuracy ^[8]. That is a harder objective to satisfy than a proxy count, because it cannot be gamed by an architecture that is cheap on paper and slow in practice. The second response changes how often the expensive part of the search has to be paid at all. Cai and colleagues named the underlying problem directly — that manually designing or running NAS “for each case” of a target device “is computationally prohibitive” — and instead trained one large “supernet” a single time, from which a specialized sub-network for a given device could be extracted “without additional training” ^[10]. That converts a cost that would otherwise recur for every device generation into a cost paid once, followed by cheap selection.

The field is not fully settled on how much of the search should be automated in the first place, and one of the strongest edge-architecture results says so in its own account of its method. Howard and colleagues built MobileNetV3 by explicitly “combining hardware-aware network architecture search (NAS) complemented by the NetAdapt algorithm” with new, manually designed architecture components ^[9] — a leading result that does not claim pure automated search was sufficient on its own. That is worth stating as a genuine, documented tension rather than resolving it: platform-aware search measurably beats hand design and beats proxy-driven search, and the strongest published edge architectures still keep a human in the loop rather than treating the search space as the final word. Beyond that tension, two limitations apply regardless of which position one takes: the search space $\mathcal{A}$ is itself a human-authored boundary on what can be discovered, and a latency- or energy-aware objective measured on one reference device is not guaranteed to hold on a different chip generation without re-measurement.

Keep the size, spend less per token

The first three strategies all shrink one dense network that must be evaluated in full for every token it produces. The fourth strategy leaves the network’s total size alone and asks a different question: can the compute spent per token be decoupled from the total capacity stored, so that most of a much larger parameter set is simply skipped for any given input?

Shazeer and colleagues stated the underlying argument for conditional computation directly: “a trainable gating network determines a sparse combination of experts to use for each example,” a mechanism they showed could scale model capacity by “over 1000x” while keeping the compute spent on any one example roughly constant ^[11]. The now-standard form of a sparse mixture-of-experts layer routes each token to a small top-$k$ subset of $E$ available experts:

with $G(x)$ a learned gating distribution over experts. Compute per token scales with the active fraction $k/E$, not with the total parameter count $N_{\text{total}}$ — the whole strategy in one line.

That decoupling is a real, documented training-cost advantage. Fedus, Zoph, and Shazeer’s Switch Transformer reported “up to 7x increases in pre-training speed with the same computational resources” relative to a dense baseline at matched compute, and a 4x speedup over the dense T5-XXL model when scaling toward a trillion parameters ^[12]. But the strategic point this article turns on is what that decoupling does not touch: memory. Jiang and colleagues state it about Mixtral 8x7B in plain terms — each token “has access to 47B parameters, but only uses 13B active parameters during inference” — and go on to say explicitly that “the memory costs for serving Mixtral are proportional to its sparse parameter count, 47B,” not the smaller active figure ^[13]. Switch Transformer documents the identical structural fact from the training side: because expert weights are split across devices, “the weights of the model increase with the number of devices,” so the full parameter set is retained somewhere even while per-token compute stays bounded ^[12].

That single fact reclassifies the whole strategy. Sparse mixture-of-experts is a serving-compute strategy, not a memory-footprint strategy, and it is not attempting to solve the problem the other three strategies exist to solve. A memory-constrained device gains nothing from routing, because what has to be resident and reachable at low latency is the full expert set, not the fraction any one token happens to use. The strategy carries its own documented costs beyond that scope mismatch. Fedus, Zoph, and Shazeer note plainly that “widespread adoption has been hindered by complexity, communication costs, and training instabilities,” with hard routing decisions introducing instability as scale increases ^[12]. Jiang and colleagues note that the routing layer itself “introduces additional overhead due to the routing mechanism and due to the increased memory loads when running more than one expert per device” ^[13] — meaning even the compute-side saving is partly offset by costs a plain dense model of the active size would not carry at all. It is worth flagging separately that Mixtral’s headline claim of outperforming or matching Llama 2 70B and GPT-3.5 across evaluated benchmarks ^[13] is Mistral AI’s own reported comparison of its own model, not an independently adjudicated result, and should be read with that qualification regardless of how the architecture itself performs.

Four axes, not one ranking

Set side by side, the four strategies do not compete on a single scale, and a comparison that reduces them to one leaderboard number is not describing what any of them actually trades off. Each holds a different quantity fixed as “already spent” and treats a different quantity as the one still to be paid:

Compression pays mostly $C_{\text{reuse}}$, a documented small fraction of a from-scratch training cost ^[3], but only if a suitable large model is available to reuse in the first place. Training small on purpose pays $C_{\text{curate}}$ and $C_{\text{train}}$ in full, with no discount, in exchange for a model that inherits nothing it was not deliberately given. Architecture search pays $C_{\text{search}}$, which can be enormous when paid fresh per target ^[7] or amortized across many targets when paid once as a supernet ^[10], on top of whatever strategy trains the architecture it discovers. Sparse mixture-of-experts is the odd one out in this accounting: its saving shows up only in $\bar c_{\text{tok}}$, the cost per served token, and it does essentially nothing to relax the constraint $M_{\text{resident}} \le B_{\text{mem}}$ that the other three strategies are built specifically to satisfy ^{[13, 12]}.

Which strategy applies is therefore a question about which term is scarce, not which strategy scores higher on a shared test. If a capable large model already exists and is reachable, compression is the fastest, cheapest route to a smaller deployable artifact, at a documented cost near a few percent of training from scratch ^[3]. If no suitable large model exists — a new domain, a deliberately restricted vocabulary, a style no available teacher has — or if even a compressed frontier-derived architecture will not fit the memory budget, training small on purpose puts the entire budget where the model will actually run, at the price of full training cost and a data-curation burden the other route avoids. If the binding constraint is a specific piece of hardware whose real latency or energy behavior cannot be predicted from any proxy, architecture search is the strategy built to measure that directly, at a search cost that is now — thanks to weight-sharing methods — often amortizable rather than paid in full for every target. And if the binding constraint is serving cost at scale, with memory to spare and enough traffic to amortize routing overhead, sparse mixture-of-experts buys quality per unit of compute that none of the memory-bound strategies can match — but it is the wrong tool entirely when the actual constraint is what fits on one device, a constraint set by physical memory and bandwidth limits that a routing decision does not relax.

None of the four is exclusive of the others in practice. MobileNetV3 already blends architecture search with manual design ^[9]; a NAS-discovered architecture can be the student in a distillation setup, or be trained with a data-efficient recipe rather than a conventional one, or be used as one expert inside a mixture-of-experts layer. Combining them does not collapse the four constraint structures into one, though — it only means a given deployed model may be answering to more than one of the four budgets in the equation above at once, which is exactly why naming which budget a specific claimed result addresses matters more than any aggregate score attached to it.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: August 2028.

One. Weight-sharing search methods that amortize the cost of architecture search across many deployment targets will displace fresh per-target search as the default for edge architectures, because the raw search cost demonstrated in early reinforcement-learning-based NAS is not a cost most teams can repeat for every new device generation. Disconfirmed if leading edge-architecture technical reports in 2028 still describe a fresh, non-amortized search run as the default path for each new target device.

Two. The line between “compress an existing model” and “train small on purpose” will blur into a shared two-stage recipe — curate a small high-quality corpus and also distill against a larger model over the same setup — rather than remaining two separate product lines, because both strategies exist to solve the same underlying problem: a small model’s difficulty learning everything it needs from a sparse raw-token signal alone. Disconfirmed if small-model technical reports in 2028 still describe compression-derived and from-scratch-trained model families as cleanly separate lines with no shared training recipe between them.

Three. Sparse mixture-of-experts will keep improving server-side serving economics without becoming a common primary on-device inference format, because its documented saving is in compute per token, not in memory footprint, and no improvement to routing changes what has to be resident on a memory-constrained device. Disconfirmed if a mainstream consumer device ships a sparse mixture-of-experts model, with its full expert set resident locally, as its primary offline inference path.

Four. Small-model releases will increasingly disclose which of these budgets produced them — a stated search cost, a stated curation cost, or an explicit “distilled from” lineage — because comparing two small models without knowing which route produced each will be recognized as no more informative than comparing two benchmark scores taken at different reasoning-effort settings. Disconfirmed if leading small-model releases in 2028 still omit any accounting of which strategy, or combination of strategies, produced the model.

What to take away

Four different engineering problems currently answer to the same phrase, “a capable small model,” and each has its own paper trail of stated rationale, quantified cost, and admitted limitation. Compression is cheap because it copies work already done, and it is only available when that work exists and is reachable. Training small on purpose pays training cost in full but answers to no inherited architecture or blind spot. Architecture search measures the real target directly, at a search cost the field has spent a decade learning to amortize rather than eliminate. Sparse mixture-of-experts solves a compute problem, not a memory problem, and treating it as a small-model strategy at all is only correct for the subset of deployments where memory was never the binding constraint. Ask which resource is actually scarce before asking which strategy is best — the second question does not have an answer independent of the first.