$$\lVert f(x) \rVert_0$$

Equation 13 · AI Research

Probing, Sparse Autoencoders, Patching, and Steering: The Main Interpretability Methods, Compared

This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.

Earlier versions of this objective penalised the code’s $\ell_1$ norm as a differentiable stand-in for sparsity, but an $\ell_1$ penalty also shrinks the magnitude of every active feature, distorting reconstruction in a way that has nothing to do with how many features are active. Rajamanoharan and colleagues introduced JumpReLU, a thresholded activation function with a learned per-feature cutoff $\theta$ trained through a straight-through gradient estimator, which lets the objective penalise the true count of active features, $\lVert$ f(x) $\rVert_0$ , directly rather than through the $\ell_1$ proxy, and reported state-of-the-art reconstruction fidelity at matched sparsity against both the earlier…

Equation guide → · Article →