Symbol k

The passage around this formula

keeping only the k largest pre-activations and zeroing the rest, and used it to train a sixteen-million-latent dictionary on GPT-4 activations over forty billion tokens, reporting metrics that improve consistently as dictionary size grows [ 8 ] . Rajamanoharan and colleagues took a different route to the same problem, keeping a continuous encoder but replacing the fixed zero threshold with a learned per-feature threshold $\theta_i$ — a feature only activates once its pre-activation clears $\theta_i$ — and report state-of-the-art reconstruction fidelity at matched sparsity on Gemma 2 activations against both the $\ell_1$ and top- k alternatives [ 9 ] .

Read this part in the article →