What does this equation mean?

Because published TopK configurations keep k in the tens to low hundreds while n runs into the millions, n $\gg$ k and the encoding term dominates almost entirely: $C_{\mathrm{SAE}}$ $\approx$ 2dnT . That single approximation explains something the paper reports without deriving: convergence — the point at which more tokens stop buying lower reconstruction error — is reached later as n grows, empirically as $\Theta$($n^{0.65}$) tokens for GPT-4-scale autoencoders [ 1 ] . Cost scales with the product of dictionary width and token count, and pushing width up forces token count up too if the dictionary is to be trained to convergence rather than merely trained. The paper is explicit that this collided with a real constraint: at their largest scale they state plainly that “because of compute constraints, we were unable to train our 16 million latent autoencoder to” the convergence frontier they used for smaller runs [ 1 ] . Sharkey and colleagues, surveying the field’s open problems, draw the economic conclusion directly: sparse dictionary learning “will probably be relatively expensive to train compared to the original model” it is being used to interpret, and that expense compounds because a separate dictionary is typically needed for every layer an investigator wants to see into [ 2 ] .