The Hardest Unsolved Problems in Mechanistic Interpretability

Where the frontier actually stops

Mechanistic interpretability’s public reputation has outrun its papers for years. Coverage says a laboratory “found the feature for deception” or “read the model’s thoughts”; the underlying report, read past the abstract, usually hedges the same claim three separate ways in its own limitations section. A companion piece in this series went through what the programme has actually established — polysemanticity, superposition in toy models, the induction-head circuit, causal intervention as the evidentiary standard — and where the gap between toy demonstrations and frontier systems currently sits. This article does not repeat that ground. It goes further out, to five specific technical problems that remain open by the documented account of the people working on them directly, not by inference from general skepticism about the field.

Each problem below is backed by a paper’s own stated limitation, not by a critic’s. That distinction matters because it changes what counts as a fair objection. It is not interesting to observe that mechanistic interpretability is hard; every field is hard at its frontier. It is interesting, and checkable, that the authors of the sparse-autoencoder scaling work say their own dictionaries do not reach a natural stopping point ^[1], that the authors of the most advanced published circuit-tracing method say they cannot draw conclusions with certainty from their own graphs ^[7], and that the authors of a widely used benchmark built an entire suite of synthetic transformers because it is difficult to validate an interpretability method against a network whose true algorithm nobody actually knows ^[6].

None of what follows is a claim that the programme has failed. Every problem here is being worked on, in public, often by the same researchers who wrote the paper documenting it. The five gaps are: whether a “feature” is reliably the right shape of thing to look for in the first place; why sparse autoencoders, the field’s dominant tool, do not finish resolving superposition even as they are scaled toward frontier models; the absence of any ground-truth circuit to check a recovered explanation against; the arbitrariness of dictionary size, under which the same activation space yields a different feature count depending on a hyperparameter nobody can currently justify from first principles; and the incompleteness of tracing a computation across the many layers of a real model, which currently produces a graph of hypotheses rather than a reading of the mechanism.

Is a feature even one-dimensional?

Almost every downstream method — sparse dictionaries, linear probes, steering vectors, activation patching read as evidence for a “direction” — assumes that a concept a model represents corresponds to a single line through activation space: present when a scalar projection is large, absent when it is small, and combined with other concepts by simple addition. That assumption is doing enormous structural work, and it is not universally true.

Engels and colleagues went looking for concepts that resist a one-dimensional description and defined, formally, what it would mean for a feature to be irreducibly multi-dimensional rather than merely two correlated one-dimensional features bundled together. Using sparse autoencoders as a discovery tool rather than a final answer, they found genuinely multi-dimensional structure in GPT-2 and Mistral 7B, including circular arrangements that represent days of the week and months of the year ^[4]. The paper’s own retitling is a small but informative fact: it now reads “Not All Language Model Features Are One-Dimensionally Linear,” a narrower and more defensible claim than the version that circulated first, because the authors found it necessary to specify exactly what kind of linearity fails.

The intervention result is the part that should unsettle a reader who treats the linear representation hypothesis as settled background. The circular day-of-week and month-of-year structures are not decorative geometry sitting alongside the computation; the authors show the model uses the circular arrangement itself to perform modular arithmetic on calendar quantities, meaning the two-dimensional shape is load-bearing for the behaviour, not an artefact of how the activations happen to be plotted. A sparse dictionary trained under the standard assumption that every feature is a single direction will, at best, represent a circular feature as several separate one-dimensional pieces glued together by correlation, which is a description of the geometry’s shadow rather than the geometry.

This is not a claim that most representations are multi-dimensional, and the authors do not make that claim either. It is a claim that the field’s central working assumption has at least one documented, intervention-verified exception, discovered using the field’s own main tool. Every method built on top of that assumption inherits an unquantified amount of the same limitation, and nobody yet has a census of how much of a frontier model’s computation runs through structure the linear hypothesis cannot see.

Superposition survives sparse autoencoders, by their own account

The original justification for sparse dictionary learning was superposition: a network with fewer neurons than the number of features it would benefit from representing can pack more features in by assigning them to overlapping, non-orthogonal directions and tolerating a controlled amount of interference. A sparse autoencoder trained on a layer’s activations is meant to reverse that packing and hand back individually interpretable directions.

The paper that introduced this method at scale is candid about how far short of “reversed” its own results fall. As the dictionary is made larger, features that looked like single, clean concepts in a small dictionary do not stay single; they split into more specific children — one reported case describes a base64-related feature in a small dictionary separating into several more particular variants once the dictionary is widened by roughly two orders of magnitude, from a few hundred learned directions to well over a hundred thousand. The authors report finding no natural stopping point in this process: nothing in the reconstruction objective tells you when to stop widening the dictionary and declare the current set of features “the” features, and the paper states plainly that sparse autoencoders reduce polysemanticity without eliminating it ^[1]. That is the method’s own account of its central limitation, published alongside its introduction, not a critique added later.

Scaling the method toward a real deployed system does not close the gap; it relocates it. Anthropic’s attribution-graph work applies a related dictionary-learning technique, cross-layer transcoders, directly to Claude 3.5 Haiku, a model in production rather than a laboratory toy, and reports that the resulting replacement model — the interpretable stand-in built from the recovered features — reproduces the original network’s behaviour closely enough to support “satisfying insight” on only about a quarter of the prompts the researchers tried ^[7]. Read alongside the feature-splitting result, the picture is consistent rather than contradictory: making the dictionary bigger buys more granular, often more interpretable pieces, but it has not been shown to buy completeness, and the clearest evidence for that comes from the researchers who did the scaling.

No ground truth to check against

Every method above needs a way to tell whether a discovered “feature” or “circuit” is the computation the network actually performs, as opposed to a pattern that merely fits the data the researcher happened to look at. For a real trained language model, no such answer key exists. Nobody has an independently verified list of the true features or the true wiring of a frontier model against which a candidate explanation can be checked, because the only thing anyone has ever had access to is the trained weights themselves — the object under interpretation is also the only available ground truth, which makes “ground truth” an odd phrase to use about it at all.

The clearest demonstration of what this absence costs comes from a paper that deliberately tests interpretability methods against out-of-distribution behaviour rather than the training distribution the method was validated on. Friedman and colleagues studied simplified proxy models — the kind produced by singular value decomposition and similar dimensionality-reduction techniques commonly used to make a network’s internals more tractable to analyse — and found consistent generalization gaps: a simplified proxy can be more faithful to the original network on in-distribution evaluation and less faithful on tests of systematic generalization, with the reverse pattern also occurring. Their own conclusion is that these results “raise questions about the extent to which mechanistic interpretations derived using tools like SVD can reliably predict what a model will do in novel situations” ^[5]. A proxy validated only where it was checked is validated only there; nothing guarantees the fit survives contact with an input the researcher did not think to try, and without ground truth there is no independent way to know how far the guarantee extends.

The field’s practical response to this absence is itself the strongest evidence that it is real. Rather than validate circuit-discovery tools against real trained models, whose true algorithms are unknown, Gupta and colleagues built InterpBench: a suite of semi-synthetic transformers trained so that their internal circuits are provably known in advance, precisely because, in the authors’ words, “it is difficult to validate such methods when the true algorithm is unknown” ^[6]. That a benchmark had to manufacture ground truth from scratch, rather than borrow it from an existing frontier model, is a direct measurement of how absent ground truth is everywhere else in the field. A method that performs well on InterpBench has passed a real test; it has not thereby demonstrated that it recovers the true algorithm of a network nobody built to have a known one.

Feature splitting and the arbitrariness of dictionary size

The splitting problem introduced above has a sharper, more recent formulation, and it points at something more troubling than “bigger dictionaries find more features.” Chanin and colleagues distinguish feature splitting, where a broad feature such as a mathematical topic divides into finer children such as algebra and geometry as the dictionary grows, from feature absorption, where a feature that looks like it should fire on a general case instead fails to fire there and the activation is picked up by a more specific child feature instead — an inconsistency in exactly which unit takes responsibility for a given input, checked across hundreds of trained sparse autoencoders. Their central negative result is that this is not simply fixed by choosing a different dictionary size or a different sparsity penalty; the paper treats absorption as a structural consequence of optimizing a sparsity objective over features that are already hierarchically related, and argues the fix has to be theoretical rather than a matter of retuning ^[2].

Leask, Nanda and colleagues then tested the assumption directly with two new techniques, and the result undercuts the idea that there is a “right” dictionary size waiting to be found by scaling further. Stitching the dictionaries of differently sized sparse autoencoders together, they show that larger dictionaries recover latents genuinely missing from smaller ones — the smaller dictionary is incomplete. Training a second, “meta” sparse autoencoder on the decoder directions of a first one, they show that what looks like a single, atomic feature in a large dictionary is itself well approximated by a sparse combination of directions from a smaller one — a reported example decomposes an “Einstein” feature into components resembling “scientist,” “Germany,” and “famous person.” Writing $d_i^{(m_2)}$ for one feature direction recovered by a dictionary of size $m_2$ and $d_j^{(m_1)}$ for the directions recovered by a smaller dictionary of size $m_1$, their finding is that

with $g$ itself sparse. A direction that a large dictionary presents as one unit is recoverable as a sparse combination of a smaller dictionary’s units, which means the large dictionary’s “atoms” are not atomic. Put the incompleteness and the non-atomicity results together and no dictionary size is privileged: a smaller one misses real structure, a larger one is decomposable into something smaller, and the paper’s own recommendation is not to keep searching for the canonical size but to pick a dictionary size pragmatically for the task at hand, because no single size is the network’s true feature set ^[3]. The number of features a sparse autoencoder reports is, on this evidence, a property of the hyperparameter chosen, not a discovery about the network.

Circuits stop at the layer wall

Even granting a fixed, useful dictionary at every layer, connecting features across layers into an end-to-end account of a computation is its own unsolved problem, and the clearest evidence is the most advanced published attempt to do it on a production model rather than a toy.

Lindsey and colleagues built attribution graphs over Claude 3.5 Haiku by tracing cross-layer transcoder features through a replacement model standing in for the original network, and they are exact about where the method’s coverage ends. The replacement model does not attempt to explain how attention patterns form; query-key computation is treated as fixed rather than interpreted, so any part of the network’s behaviour that depends on how attention gets allocated is outside the method’s scope by construction. The graphs also carry explicit “error nodes” recording the gap between the replacement model’s activations and the real network’s, which stand in for whatever fraction of the computation the recovered features fail to reconstruct — a bookkeeping category for the parts of the mechanism nobody has explained yet, rather than an explanation of them. And across the graphs the authors did produce, they report that their method yields what they call satisfying insight on only about a quarter of the prompts attempted, with many others left effectively unusable specifically because of the frozen-attention limitation ^[7].

The authors’ own epistemic framing is the most important sentence in the paper for a piece about unsolved problems, and it is worth quoting rather than summarizing: they state that attribution graphs cannot be used to draw conclusions with certainty about the underlying model, because the graphs supply hypotheses about the mechanisms operating inside it rather than a direct readout of those mechanisms ^[7]. That is a categorical distinction, not a hedge. A hypothesis can be well-supported, can survive intervention, can generalize across many prompts — the induction-head result establishes exactly that kind of case at small scale — but a graph produced by an incomplete replacement model, with attention held fixed and a bounded fraction of the activity assigned to unexplained residue, has not yet crossed into being a reading of the computation. The gap between “we traced a plausible causal graph” and “we explained what the layers are doing to each other” is precisely where cross-layer tracing at frontier scale currently sits, and it is the paper’s own laboratory that says so.

What would even count as done?

Suppose every problem above were solved: features that are always the right shape, dictionaries at a principled size, ground truth to check against, complete cross-layer graphs. There remains a question the field has not settled even in principle — what threshold of evidence makes an explanation of a network component count as finished rather than merely plausible.

Sharkey and colleagues, in the field’s own forward-looking review of its open problems, put this plainly: it is often hard to distinguish a faithful explanation of a network component from one that is merely plausible, and the paper traces this back to a lack of adequate theory for how to decompose a network into the right atomic units in the first place, noting broad agreement among researchers that this remains a central unresolved problem rather than a solved one with rough edges ^[9]. That is a strong claim to find in a field’s own consensus document: not that a particular technique needs refinement, but that the field does not yet agree on the criterion an explanation must meet to stop being a hypothesis.

Olah’s earlier and more informal statement of the same gap is worth setting alongside it because it shows the uncertainty is not new. Sketching an ambitious long-run vision for the field — that claims about a whole model’s behaviour might eventually be built up from claims about its circuits — the essay is explicit that the vision is aspirational rather than demonstrated, closing on the frank admission that it is far from clear any of it will ultimately be possible ^[8]. Three years and a great deal of empirical progress separate that statement from the Sharkey review, and the specific problem it names — what actually counts as validated, as opposed to merely consistent with the evidence gathered so far — has not moved from aspiration to settled method in the interval. Every other problem in this article is, in a sense, downstream of this one: without an agreed bar for what a satisfying explanation has to survive, disputes about whether a given feature or circuit paper has actually shown what it claims have no fixed way to be resolved.

Predictions, with what would falsify them

These are forecasts, clearly separated from the sourced findings above. Horizon: 12 August 2029. Assumptions: no change in the dominant transformer architecture, continued publication of interpretability work by frontier laboratories at a pace comparable to the past three years, and no single method achieving field-wide consensus as a solution to more than one of the five problems above.

One. Feature-absorption and canonical-unit critiques will shift the field’s standard reporting away from a single headline dictionary size toward reporting results across multiple sizes or toward architectures designed to be stitched. Disconfirmed if a majority of widely cited sparse-autoencoder papers in 2029 still report results from one fixed dictionary size with no stability or stitching analysis.

Two. Cross-layer attribution methods will extend to explain attention formation directly, rather than holding query-key computation fixed, because the frozen-attention limitation is the single most-cited blocker in the current published account. Disconfirmed if leading circuit-tracing papers in 2029 still treat attention patterns as given rather than as a target of explanation.

Three. Synthetic-ground-truth benchmarks in the style of InterpBench will multiply and diversify by task rather than be superseded, because the underlying problem — no real model has a known true circuit — is structural rather than a temporary tooling gap. Disconfirmed if a method for establishing ground truth inside an actual frontier model, rather than a purpose-built synthetic one, becomes standard practice.

Four. No single, field-wide definition of “satisfying mechanistic explanation” will be adopted by the 2029 horizon; instead, the field will continue to operate with goal-relative bars — one standard for auditing, a looser one for a usable steering handle, a stricter one for a publishable circuit claim. Disconfirmed if a specific faithfulness criterion becomes the uncontested field standard cited by default across auditing, steering and circuit-discovery papers alike.

What to take away

Established, and not repeated in doubt here: sparse dictionaries recover more interpretable directions than raw neurons, cross-layer attribution graphs produce real, checkable hypotheses about production models, and the field’s own practitioners are unusually candid, in print, about where their methods stop. Not established: that a feature is reliably a one-dimensional direction, that a sparse dictionary’s size converges on anything the network itself would recognize as its true feature count, that any recovered explanation of a frontier model’s computation has been checked against a known-correct answer rather than against another guess, that cross-layer tracing currently produces more than partial, hypothesis-grade coverage of a real model’s behaviour, and that the field has an agreed test for when an explanation is finished.

The honest summary is not that mechanistic interpretability has stalled. It is that the frontier has moved to exactly the five places listed here, each identified by the researchers who ran into it rather than by an outside critic, and each still open the way its own paper says it is open.