≈

The passage around this formula

Its intellectual foundation is the same scaling-law literature that shaped how large models are trained, applied in the opposite direction. Hoffmann and colleagues showed that contemporary large language models had been trained on too little data relative to their parameter count, and that for a fixed training budget, model size and training tokens should grow in roughly equal proportion — “for every doubling of model size the number of training tokens should also be doubled” [ 4 ] . The underlying loss law is commonly written as $L(N, D) \approx E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$. with N parameters and D training tokens. Chinchilla’s question was: given a fixed compute budget C , what N and D minimize L ? The…

Read this part in the article →