Equation 8 · Shrink It, Train It Small, or Search for It: The Main Strategies for Small Models, Compared
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
This equation gives an approximation: it relates the quantities while allowing an approximation. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
Symbol L
L is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Symbol D
D occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.
Symbol E
E is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Symbol A
A occurs above the fraction bar. The numerator is divided by the entire denominator below it.
Symbol N^α
N^α occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.
Symbol B
B occurs above the fraction bar. The numerator is divided by the entire denominator below it.
Symbol D^β
D^β occurs below the fraction bar. The quantity above the bar is divided by this expression; zero is excluded as a denominator.
superscript
A raised number can be a power. When it is a label or bound, it selects a case or the upper limit of a sum; the formula’s structure distinguishes these uses.
See an illustrated explanation →How to interpret it
With a fixed numerator, increasing a nonzero denominator reduces the fraction. Its accuracy depends on the assumptions and range of use described in the article.
What the article says around this equation
Its intellectual foundation is the same scaling-law literature that shaped how large models are trained, applied in the opposite direction. Hoffmann and colleagues showed that contemporary large language models had been trained on too little data relative to their parameter count, and that for a fixed training budget, model size and training tokens should grow in roughly equal proportion — “for every doubling of model size the number of training tokens should also be doubled” [ 4 ] . The underlying loss law is commonly written as . with N parameters and D training tokens. Chinchilla’s question was: given a fixed compute budget C , what N and D minimize L ? The…
Read the full surrounding passage
Its intellectual foundation is the same scaling-law literature that shaped how large models are trained, applied in the opposite direction. Hoffmann and colleagues showed that contemporary large language models had been trained on too little data relative to their parameter count, and that for a fixed training budget, model size and training tokens should grow in roughly equal proportion — “for every doubling of model size the number of training tokens should also be doubled” [ 4 ] . The underlying loss law is commonly written as . with N parameters and D training tokens. Chinchilla’s question was: given a fixed compute budget C , what N and D minimize L ? The data-efficient small-model strategy asks a different question with the same law: given a fixed, small N set by the deployment target, what choice of D — and, crucially, what quality of D — minimizes L ? Fixing the small side of the equation first and spending the freed budget on data rather than on parameters is the strategy’s entire premise.
Sources cited in the surrounding passage
These citations give research context. Read each source to check which claims it supports.