What does this equation mean?

Leask, Nanda and colleagues then tested the assumption directly with two new techniques, and the result undercuts the idea that there is a “right” dictionary size waiting to be found by scaling further. Stitching the dictionaries of differently sized sparse autoencoders together, they show that larger dictionaries recover latents genuinely missing from smaller ones — the smaller dictionary is incomplete. Training a second, “meta” sparse autoencoder on the decoder directions of a first one, they show that what looks like a single, atomic feature in a large dictionary is itself well approximated by a sparse combination of directions from a smaller one — a reported example decomposes an “Einstein” feature into components resembling “scientist,” “Germany,” and “famous person.” Writing $d_i^{(m_2)}$ for one feature direction recovered by a dictionary of size $m_2$ and $d_j^{(m_1)}$ for the directions recovered by a smaller dictionary of size $m_1$ , their finding is that $d_i^{(m_2)} \approx \sum_{j=1}^{m_1} g_j\, d_j^{(m_1)}, \qquad \lVert g \rVert_0 \ll m_1, \quad m_1 < m_2$. with g itself sparse. A direction that a large dictionary presents as one unit is recoverable as a sparse combination of a smaller dictionary’s units, which means the large dictionary’s “atoms” are not atomic. Put the incompleteness and the non-atomicity results together and no dictionary size is privileged: a smaller one misses real structure, a larger one is decomposable into something smaller, and the paper’s own recommendation is not to keep searching for the canonical size but to pick a dictionary size pragmatically for the task at hand, because no single size is the network’s true feature set [ 3 ] . The number of features a sparse autoencoder reports is, on this evidence, a property of the hyperparameter chosen, not a discovery about the network.