What does this equation mean?

The $\ell_1$ penalty has a known cost: it does not directly control how many dictionary elements fire, only how much their combined magnitude is discouraged, and it systematically shrinks the elements that do fire toward zero, biasing the reconstruction. Two later refinements address this more directly. Gao and colleagues introduced k -sparse encoding, which drops the tunable penalty in favour of a fixed sparsity budget enforced structurally, $z = \mathrm{TopK}_k\!\left(W_{\mathrm{enc}}(a - b_{\mathrm{dec}})\right)$. keeping only the k largest pre-activations and zeroing the rest, and used it to train a sixteen-million-latent dictionary on GPT-4 activations over forty billion tokens, reporting metrics that improve consistently as dictionary size grows [ 8 ] . Rajamanoharan and colleagues took a different route to the same problem, keeping a continuous encoder but replacing the fixed zero threshold with a learned per-feature threshold $\theta_i$ — a feature only activates once its pre-activation clears $\theta_i$ — and report state-of-the-art reconstruction fidelity at matched sparsity on Gemma 2 activations against both the $\ell_1$ and top- k alternatives [ 9 ] .