Symbol C_SAE

The passage around this formula

…model for training the dictionary is $C_{\mathrm{SAE}} \;\approx\; 2\,d\,(n + k)\,T$. Because published TopK configurations keep k in the tens to low hundreds while n runs into the millions, n $\gg$ k and the encoding term dominates almost entirely: $C_{\mathrm{SAE}}$ $\approx$ 2dnT . That single approximation explains something the paper reports without deriving: convergence — the point at which more tokens stop buying lower reconstruction error — is reached later as n grows, empirically as $\Theta$($n^{0.65}$) tokens for GPT-4-scale autoencoders [ 1…

Read this part in the article →