addition

The passage around this formula

with N parameters and D training tokens, the factor of six counting the forward and backward passes. Empirically, test loss falls as a power law in each of parameters, data, and compute over many orders of magnitude, a relationship first characterised systematically by Kaplan and colleagues [ 5 ] . The important structural feature is the functional form: a term of the shape $L(N) \approx L_\infty + \left(\frac{N_c}{N}\right)^{\alpha_N}$. has an irreducible floor $L_\infty$ and diminishing returns above it. Doubling N buys a fixed decrement in loss, not a fixed multiple of capability.

Read this part in the article →