What does this equation mean?

By this point compute had become the binding constraint, and the field did not know how to spend it. Kaplan and colleagues characterised the relationship empirically, finding that cross-entropy loss scales as a power law in model size, dataset size and compute, with trends spanning more than seven orders of magnitude, and concluding that compute-efficient training meant very large models on relatively modest data, stopped well short of convergence [ 16 ] . The functional form is the important part: $L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}$. Hoffmann and colleagues then trained over four hundred models and reached a different allocation: model size and training tokens should scale in roughly equal proportion, and the large models of the era were significantly undertrained. Chinchilla, at 70 billion parameters trained on roughly four times more data than the 280-billion-parameter Gopher at the same compute budget, outperformed it and reached 67.5% on MMLU [ 17 ] .