Symbol A

The passage around this formula

…how large models are trained, applied in the opposite direction. Hoffmann and colleagues showed that contemporary large language models had been trained on too little data relative to their parameter count, and that for a fixed training budget, model size and training tokens should grow in roughly equal proportion — “for every doubling of model size the number of training tokens should also be doubled” [ 4 ] . The underlying loss law is commonly written as $L(N, D) \approx E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$. with N parameters and D training…

Read this part in the article →