≈

The passage around this formula

The architecture underneath a Claude model, like nearly every other frontier language system, is a decoder-only transformer trained to predict the next token over a very large corpus [ 12 ] . What that training actually produces is not knowledge in any curated sense but a prior — a broad distribution over plausible continuations that encodes an enormous amount about language, code, and the structure of arguments, and comparatively little about what any particular operator wants done with it. Two variables set the price of that prior. Pretraining compute for a dense transformer is well approximated by $C_{\mathrm{pre}} \approx 6ND$. where N is the parameter count and D is the number of training tokens,…

Read this part in the article →