Equation 7 · Part 1 · What Interpretability Actually Costs to Do at Scale
Symbol k
What this part means
the because published topk configurations keep.
Its job in the formula
k is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Full expression→Symbol k→Article meaning
Where the article explains it
Because published TopK configurations keep k in the tens to low hundreds while n runs into the millions, n k and the encoding term dominates almost entirely: 2dnT .
The passage around this formula
and the specific variant OpenAI’s interpretability team used to push this to frontier scale, the TopK autoencoder, replaces the soft penalty with an explicit constraint: exactly k latents fire, chosen by magnitude, and the rest are hard-zeroed. Gao and colleagues introduced this variant, established scaling laws relating autoencoder size and sparsity to reconstruction error, and — this is the operative fact for a cost accounting — trained a sixteen-million-latent autoencoder on GPT-4’s activations over forty billion tokens [ 1 ] .
Learn the underlying idea
A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.
Open the illustrated variables: a letter stands for a value guide →
See this notation across published equations →
Sources cited in the surrounding passage
These citations provide research context; check each source for the exact claim it supports.