What does this equation mean?

Work out what that architecture actually spends compute on per token, because the two halves of it behave differently. The encoding step needs a score for every one of the n candidate latents before it can select the top k , so it is an unavoidably dense matrix multiply: roughly 2dn floating-point operations. The decoding step only touches the k latents that survived, so it is sparse: roughly 2dk operations. Summed and multiplied across T training tokens, a first-order compute model for training the dictionary is $C_{\mathrm{SAE}} \;\approx\; 2\,d\,(n + k)\,T$. Because published TopK configurations keep k in the tens to low hundreds while n runs into the millions, n $\gg$ k and the encoding term dominates almost entirely: $C_{\mathrm{SAE}}$ $\approx$ 2dnT . That single approximation explains something the paper reports without deriving: convergence — the point at which more tokens stop buying lower reconstruction error — is reached later as n grows, empirically as $\Theta$($n^{0.65}$) tokens for GPT-4-scale autoencoders [ 1 ] . Cost scales with the product of dictionary width and token count, and pushing width up forces token count up too if the dictionary is to be trained to convergence rather than merely trained. The paper is explicit that this collided with a real constraint: at their largest scale they state plainly that “because of compute constraints, we were unable to train our 16 million latent autoencoder to” the convergence frontier they used for smaller runs [ 1 ] . Sharkey and colleagues, surveying the field’s open problems, draw the economic conclusion directly: sparse dictionary learning “will probably be relatively expensive to train compared to the original model” it is being used to interpret, and that expense compounds because a separate dictionary is typically needed for every layer an investigator wants to see into [ 2 ] .