What a Circuit Explains: The State and Limits of Mechanistic Interpretability

The claim a circuit makes

Mechanistic interpretability is the attempt to describe what a trained neural network computes in terms a person can hold in mind, and to check that description against the network rather than against intuition. The programme was stated most cleanly in Distill in 2020, in three claims its authors were careful to label speculative: that networks contain features, meaningful directions or units of computation; that features are wired into circuits by the weights between them; and that analogous features and circuits recur across models trained on different data, a property called universality ^[1]. That essay’s worked examples came from a vision network — curve detectors, a dog-head circuit, and neurons that fired for both cat faces and car fronts.

Six years later, popular coverage routinely reports the programme’s conclusion as though it were finished: that we can now read what a model is thinking, that we have found the feature for deception, that a circuit has been extracted and therefore understood. Almost none of that survives contact with the papers, which are in general considerably more careful than their summaries.

This article separates what has been established from what is being assumed. The distinction is not pedantic. Interpretability results are increasingly cited in arguments about deployment and oversight, and an argument that rests on an overstated result inherits its weakness silently.

The organising image is a watch movement on a bench. You cannot learn what a bridge was holding by looking at the assembled watch; you learn it by lifting the bridge and seeing what moves. That is a real method, and it is the method interpretability actually uses. It also has real limits: the answer is about the movement in front of you, the act of lifting changes the thing you are measuring, and a tray of sorted parts is not yet an explanation of timekeeping.

What a feature is taken to be

The load-bearing assumption underneath most current work is the linear representation hypothesis: that a human-legible concept corresponds to a direction in some activation space, that the presence of the concept corresponds to a large component along that direction, and that concepts compose roughly additively. This is why probing works when it works, why adding a vector to the residual stream can steer behaviour, and why the whole vocabulary of “feature directions” makes sense at all.

The hypothesis is often asserted loosely; Park, Choe and Veitch gave it a usable formal treatment. They distinguish two senses of linear representation — one in the output space, connected to linear probing, and one in the input space, connected to steering — and show that unifying them requires abandoning the naive Euclidean geometry of the activation space in favour of a causal inner product that respects the structure of the concepts being represented, with experiments on LLaMA-2 supporting the construction ^[5].

Two things are worth marking about this. First, it is a hypothesis with substantial supporting evidence, not a theorem, and it is stated by its careful proponents as such. Second, even in its formalised version it says that some concepts are represented linearly under a particular geometry. It does not say that all of a model’s computation is a sum of concept directions, and no result establishes that.

Why a neuron is not a feature

The obvious candidate for a feature is a neuron, and the obvious candidate fails. Individual neurons in trained networks fire on collections of inputs with no common human description — a neuron responds to legal language and to a particular Python idiom and to nothing else identifiable. This is polysemanticity, and it was documented in vision networks in the original circuits work ^[1] and in language models shortly after.

Bolukbasi and colleagues then showed something sharper and more uncomfortable: individual neurons in BERT, and linear combinations of them, can appear to encode a single simple concept when they are in fact encoding something considerably more complex. They traced the effect to two causes — geometric properties of the embedding space, and the narrowness of the corpora on which interpretations are typically checked — and drew the methodological conclusion that an interpretation validated on one dataset is not validated ^[6]. That paper is from 2021, and the failure mode it names has not gone away.

The dominant explanation for polysemanticity is superposition: a network that needs to represent more sparsely-occurring features than it has dimensions can do so by assigning features to non-orthogonal directions, accepting a small amount of interference in exchange for representing many more things. Elhage and colleagues studied this in toy models simple enough to be understood completely, and found that superposition is not a pathology but an optimum — that networks prefer to encode features directly at low sparsity and shift into superposed encodings as feature sparsity rises, with distinct phase-like transitions between regimes and geometric structure in how the superposed directions arrange themselves ^[4].

The status of that result should be stated precisely, because it is frequently overstated. In toy models, superposition is demonstrated: the ground-truth features are known because the experimenter created them, and the network’s use of them can be read off. In real language models, superposition is an inference to the best explanation of polysemanticity. It is a good explanation, it makes correct predictions, and it is not a direct observation. Nobody has enumerated the true feature set of a frontier model and shown it exceeds the model’s dimensionality, because nobody has enumerated the true feature set of any frontier model.

What a dictionary certifies, and what it does not

If features are directions and neurons are not the directions, the natural move is to look for the directions directly. Sparse dictionary learning does this: train an overcomplete autoencoder on a model’s activations whose hidden layer is pushed toward sparsity, and read its dictionary elements as candidate features. Cunningham and colleagues showed this produces features substantially more interpretable than alternatives such as principal components or individual neurons, and — importantly — that the recovered features can be used for finer-grained causal identification of the components responsible for a specific behaviour than prior methods achieved ^[7].

The method scales in the engineering sense. Gao and colleagues introduced k-sparse autoencoders to control sparsity directly, established clean scaling laws relating autoencoder size and sparsity to reconstruction, and trained a sixteen-million-latent autoencoder on GPT-4 activations over forty billion tokens ^[8]. At comparable ambition, up to thirty-four million features were trained on the middle-layer residual stream of Claude 3 Sonnet, yielding features that respond to abstract concepts and generalise across languages and modalities ^[9].

Now the careful part. Consider what the training objective actually asks for. Writing $x$ for an activation vector, $f(x)$ for the sparse code and $\hat{x}$ for the reconstruction, the objective has the form

Every term refers to the activation vector. No term refers to what the model does with that activation afterwards. The objective rewards a code that reconstructs the activation sparsely; it is indifferent to whether the dictionary elements correspond to anything the network’s downstream layers treat as a unit. Low reconstruction error at high sparsity is therefore evidence that the activation distribution is sparsely decomposable in the trained basis. It is not evidence that the basis is the model’s own.

Three empirical results sharpen this into something more than a philosophical caveat.

Interpretability scores do not discriminate trained networks from random ones. Heap and colleagues trained sparse autoencoders on randomly initialised transformers across a range of Pythia sizes and found that automated interpretability scores and reconstruction metrics came out similar to those from trained models ^[10]. A metric that assigns comparable scores to a network that has learned language and a network that has learned nothing is not measuring what it is being used to claim.

Downstream usefulness has not been demonstrated to beat simple baselines. Kantamneni and colleagues ran what is close to a decisive test on one concrete task, sparse probing, under four regimes chosen to favour a good feature basis — data scarcity, class imbalance, label noise, and covariate shift. Sparse autoencoders occasionally beat baselines on individual datasets, but the authors report they “are unable to design ensemble methods combining SAEs with baselines that consistently outperform ensemble methods solely using baselines” ^[11]. That is a negative result on one application, not a refutation of the method; it is also exactly the kind of comparison the field had mostly not been running.

The practitioners say so themselves. The Claude 3 Sonnet work states plainly that “our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations” ^[9]. When a laboratory’s own paper says this and the coverage of that paper says the model’s concepts have been catalogued, the gap is in the coverage.

Circuits, and the one case that is genuinely nailed down

Features alone are an inventory. A circuit is a claim about how features are combined, and that requires knowing what the architecture can express.

The mathematical framework for transformer circuits supplied that. It decomposes attention-only transformers into tractable pieces: a zero-layer model captures bigram statistics readable directly from the weights; a one-layer attention-only model behaves as an ensemble of bigram and skip-trigram models; and at two layers, composition between attention heads becomes possible and qualitatively new algorithms appear. The decomposition of each head into a query-key circuit, which decides where attention goes, and an output-value circuit, which decides what is written when it gets there, is the piece of formal apparatus most subsequent work depends on ^[2].

The clearest worked case in the field remains the induction head: a pair of composed heads that finds an earlier occurrence of the current token and copies what followed it, implementing “if A was followed by B before, predict B after A now”. What makes it the clearest case is not the mechanism, which is simple, but the number of independent lines of evidence brought to bear. Olsson and colleagues assembled six: induction heads form during a narrow phase change in training that coincides with a sharp improvement in in-context learning; architectural modifications that shift when induction heads can form shift the in-context learning improvement to match; direct ablation of the heads in small models substantially degrades in-context learning; the heads implement recognisably general behaviours including copying and translation; the mechanism in small models generalises naturally to broader cases; and the relevant behaviours vary continuously from small to large models. The study covered thirty-four transformers across training and over fifty thousand attention-head ablations ^[3].

That is what a well-supported mechanistic claim looks like, and it is worth noting how much work it took. The authors were also explicit that the sixth strand — the extrapolation from small models, where the mechanism is verified, to large ones, where it is not — is the weakest.

The second reference case is the indirect object identification circuit in GPT-2 small: twenty-six attention heads sorted into seven functional classes, identified by causal intervention and evaluated against three explicit criteria the authors defined — faithfulness, completeness, and minimality ^[12]. Those criteria matter more than the circuit. Before them, “we found a circuit” had no failure condition attached.

Circuit-finding has since been partly automated. Conmy and colleagues built ACDC, which prunes edges of the computational graph by patching, and validated it by recovery: on GPT-2 small it rediscovered all five component types of a previously hand-analysed Greater-Than circuit, and selected sixty-eight of the model’s thirty-two thousand edges, all of which had been found manually by earlier work ^[13]. Recovery of known answers is a legitimate validation and a limited one — it shows the tool does not miss what humans found, not that it finds what humans missed.

Intervention is the standard, and it is not self-certifying

The methodological centre of the field is the shift from correlational to interventional evidence. A neuron that fires whenever the input concerns France has told you nothing about whether the model uses it. The test is to change it and see what happens.

The standard instrument is activation patching: run the model on a clean input, run it on a corrupted one, then substitute the activations of a chosen component from one run into the other and measure the change in some behavioural metric. Formally, for a set of components $\mathcal{C}$ and a metric $m$,

Meng and colleagues used a version of this — causal tracing — to localise factual recall, finding that a distinct set of steps in middle-layer feed-forward modules mediate factual predictions at the subject token, and then used the localisation to perform rank-one weight edits that changed specific facts ^[14]. The edit is the strongest form of the argument: a localisation claim that supports a successful targeted modification has done more than describe.

Geiger and colleagues have argued that all of this belongs to a single formal framework, causal abstraction, in which an interpretation is a high-level causal model claimed to be a faithful simplification of the low-level network, and in which activation patching, causal mediation analysis, circuit analysis and sparse autoencoders all appear as special cases ^[17]. This is the most promising route to making “faithful” a technical rather than rhetorical term, and it is a research programme rather than a settled foundation.

Two results establish that intervention does not certify itself.

Zhang and Nanda examined the methodological choices inside patching — which metric, which corruption, which direction — and found that they can produce substantially different conclusions about localisation and circuit membership for the same behaviour in the same model ^[15]. A localisation claim is therefore a claim relative to a protocol, and a paper that does not report the protocol has not reported the result.

Makelov, Lange and Nanda demonstrated a harder failure. When patching a subspace changes the output in the expected way, the natural reading is that the subspace carries the feature. They showed the same behavioural change can be produced by activating a dormant parallel pathway that has nothing to do with the model’s normal computation, and exhibited the effect on both indirect object identification and factual recall — real tasks, not constructed counterexamples ^[16]. Behavioural change under intervention is necessary evidence for a mechanistic claim, and it is not sufficient.

The scalability gap

Everything above is easier to state than to do at frontier scale, and the gap is the field’s central practical problem.

The most advanced published attempt to trace computation in a production model builds a replacement model out of cross-layer transcoder features, freezes attention patterns, and constructs attribution graphs whose nodes are active features and whose edges are linear effects between them. Its authors report the replacement model matches the original’s output on roughly half of diverse prompts, and list the limitations without hedging: the method does not attempt to explain how attention patterns are formed, so query-key dynamics are outside its scope; reconstruction error nodes stand in for a portion of the computation that goes unexplained; perturbation experiments show effects whose direction correlates around 0.8 with predictions but whose magnitudes deviate increasingly across layers; the resulting graphs are frequently very complex and hard to understand; and there are many cases where the attention limitation renders a graph, in the authors’ own word, essentially useless ^[19].

That is an honest account of a genuine advance. It is also nothing like a reading of the model’s cognition. Read alongside sparse feature circuits, which do produce interpretable causal subgraphs over features and support a downstream application — improving a classifier’s generalisation by ablating features a human judges irrelevant to the task — the picture is of methods that work partially, on selected behaviours, with known blind spots ^[18].

The gap has three components that should not be conflated. The coverage gap: methods explain a fraction of the computation and the rest is bookkeeping. The verification gap: at toy scale ground truth exists, at frontier scale it does not, so faithfulness must be estimated by proxies whose own validity is contested. The labour gap: the induction-head result required thirty-four models and fifty thousand ablations for one mechanism, and frontier models have vastly more behaviours than anyone will ever have analyst-hours to trace.

What is the explanation for?

The most useful question to put to any interpretability method is not whether it works but what it is for, because auditing, prediction and control have different success conditions, and a method can satisfy one and fail the others.

Auditing asks whether a model has a property it should not — a hidden objective, a backdoor, an undisclosed behaviour. The most informative evidence here comes from a deliberate test: a model was trained with a concealed objective, and four teams were set to find it blind. Three of the four succeeded, using a mix of methods — interpretability with sparse autoencoders, behavioural attacks, and training-data analysis — and eight auditing techniques were subsequently evaluated ^[20]. Read carefully, this is encouraging and modest at once. Interpretability contributed. It was not uniquely decisive, and access to training data was among the techniques available. Auditing demands coverage — a guarantee about what is not there — and coverage is exactly what current feature dictionaries do not have.

Control asks whether the model’s behaviour can be reliably changed by acting on internals. This is the criterion current methods meet best, and notably it does not require a complete explanation. Arditi and colleagues found that refusal in chat models is mediated to a large extent by a single direction in the residual stream: erasing it prevents refusal of harmful instructions, adding it induces refusal of harmless ones, and the effect held across thirteen open-source chat models up to seventy-two billion parameters ^[21]. That is a strong, replicated, actionable finding — and it is a handle, not an account of why refusal is organised that way. Handles are valuable. They are also dual-use, as the paper’s own jailbreak demonstration makes explicit.

Prediction asks whether internals let you say what the model will do on inputs you have not tried. This is the weakest of the three and the one most often implied. No published method reliably predicts out-of-distribution behaviour from internal structure; the field’s own agenda-setting review places methods, application to specific goals, and socio-technical questions as three distinct categories of open problem, which is itself an admission that the first is not yet solved ^[22].

Where the disagreement actually lies

The field is not divided into believers and sceptics; the serious disagreements are narrower and more interesting.

The first is ontological: whether “features” are the right unit at all, or an artefact of the methods used to look for them. Sparse dictionary learning presupposes that activations decompose into a sparse sum of atoms. If the underlying structure is instead curved, or better described as a low-dimensional manifold, or genuinely distributed in a way no sparse basis captures, then dictionaries would still fit well and still mislead.

The second is evidentiary: what a faithfulness metric is worth. One camp holds that behavioural equivalence under intervention is the operative definition and improvable by better protocols; another holds, with the subspace-illusion result behind them, that behavioural equivalence is systematically confusable with mechanism ^{[16, 17]}.

The third is strategic: whether to scale existing methods to frontier models now, accepting known unfaithfulness, or to establish rigorous foundations at small scale first. Both positions are held by capable researchers with the same evidence in front of them. This one is not resolvable by argument; it will be settled by whichever branch produces a result the other cannot match.

Predictions, with what would falsify them

These are forecasts and clearly marked as such. Horizon: 8 August 2029. Assumptions: no change in the dominant transformer architecture, continued publication of interpretability work by frontier laboratories, and no regulatory mandate forcing disclosure of internals.

One. Interpretability will be adopted operationally for control long before it is adopted for auditing, because control needs a working handle and auditing needs coverage. Indicators: steering-vector and feature-ablation techniques appearing in production safety stacks; audit reports still resting primarily on behavioural evaluation. Disconfirmed if a published third-party audit certifies the absence of a property in a frontier model on primarily mechanistic evidence.

Two. Reconstruction-based metrics will be displaced as the headline measure of a feature dictionary by downstream-task and intervention-based evaluations, following the pattern the random-transformer and sparse-probing results have set. Disconfirmed if the standard reported figures for dictionary methods in 2029 remain reconstruction loss and sparsity with no mandatory downstream comparison.

Three. The count of frontier-model behaviours with an end-to-end circuit-level account meeting the faithfulness, completeness and minimality bar will grow slowly — by small multiples, not orders of magnitude — because the binding constraint is analyst labour and verification difficulty rather than tooling. Disconfirmed if automated pipelines produce validated circuits for hundreds of distinct frontier behaviours.

Four. Some prominent published feature or circuit interpretation will be shown to have been an illusion of the kind already documented, and the correction will be less widely reported than the original. Disconfirmed if no such retraction or substantive correction appears by the horizon.

What to take away

Established: neurons are frequently polysemantic; superposition explains this well and is demonstrated in toy models; sparse dictionaries recover more interpretable directions than neurons or principal components; induction heads exist, form in a training phase change, and causally support in-context learning in small models; the indirect object identification circuit is real and was found by intervention; single directions can mediate substantial behaviours such as refusal across many models; causal intervention beats correlation as evidence.

Not established: that recovered dictionary features are the model’s own units; that any frontier model’s computation has been explained in more than fragments; that behavioural change under intervention licenses a mechanistic claim; that internals support prediction of out-of-distribution behaviour; that interpretability can currently certify the absence of a property.

The bridge is genuinely off the movement, and the wheel beneath it has genuinely started to lean. That tells you something true, and it tells you about this movement, on this bench, under this light. The distance between that and understanding the watch is the distance the field still has to travel, and the papers are considerably more candid about it than the coverage.