≈

The passage around this formula

Because published TopK configurations keep k in the tens to low hundreds while n runs into the millions, n $\gg$ k and the encoding term dominates almost entirely: $C_{\mathrm{SAE}}$ $\approx$ 2dnT . That single approximation explains something the paper reports without deriving: convergence — the point at which more tokens stop buying lower reconstruction error — is reached later as n grows, empirically as $\Theta$($n^{0.65}$) tokens for GPT-4-scale autoencoders [ 1 ] . Cost scales with the product of dictionary width and token count, and pushing width up forces token count up too if the dictionary is to be trained to convergence rather than merely trained. The paper is explicit that this collided…

Read this part in the article →