≈

The passage around this formula

Work out what that architecture actually spends compute on per token, because the two halves of it behave differently. The encoding step needs a score for every one of the n candidate latents before it can select the top k , so it is an unavoidably dense matrix multiply: roughly 2dn floating-point operations. The decoding step only touches the k latents that survived, so it is sparse: roughly 2dk operations. Summed and multiplied across T training tokens, a first-order compute model for training the dictionary is $C_{\mathrm{SAE}} \;\approx\; 2\,d\,(n + k)\,T$. Because published TopK configurations keep k in the tens to low hundreds while n runs into the millions, n $\gg$ k and the encoding term dominates almost…

Read this part in the article →