The wall exists because the story is always available
Mechanistic interpretability’s central promise is that a trained network’s computation can be taken apart and described in terms a person can check, not merely narrate. That promise is precisely why the field is unusually careful about method compared with much of deep learning: its best papers state explicit criteria for what counts as a circuit, publish ablation counts in the tens of thousands, and describe their own feature dictionaries as incomplete. The seriousness of the best work is exactly why the ten failure modes catalogued below matter. They are not the mistakes of a careless field. They are the mistakes that survive careful, well-intentioned researchers working inside a genuinely hard measurement problem, because the network under study is enormous, the space of stories compatible with any one observed activation pattern is larger still, and the step that would rule most of those stories out — an independent, adversarial rerun — is the step that is hardest to fund, hardest to referee, and easiest to skip.
Every failure mode below has already occurred in a documented, citable instance: a claim that read as solid until someone tried to break it, tried to reproduce it, or ran a comparison nobody had run yet. None of them requires bad faith. Several are close to invisible from inside a single paper and only surface once a second team, a held-out dataset, or a systematic audit is put in the room with the original result. That is the discipline an audit room exists to enforce, and it is the organizing image this article borrows: not the interpretability lab where circuits are found, but the room next door where a claim about a circuit gets checked before anyone is asked to believe it.
The story that fit because it was built to fit
The easiest way to explain a component in a neural network is to already know, before looking, what you expect it to do. Circuit-hunting invites this directly: an analyst forms a hypothesis about a behavior, goes looking in the model for a component whose activation pattern is compatible with that hypothesis, and — because a modern network has thousands of components and any one behavior admits many superficially plausible causal stories — usually finds one. The field’s founding methodological statement was candid about this risk from the outset: the original circuits program at Distill explicitly labelled its central claims — that networks contain features, that features are wired into circuits, that circuits recur across models — as “speculative,” not established [1].
The empirical demonstration that the risk is real, not merely theoretical, came from a study of neuron interpretations in BERT. Tolga Bolukbasi and colleagues showed that individual neurons, and simple linear combinations of them, can appear to encode one clean, human-legible concept when the underlying computation is considerably messier, and traced the illusion to two specific causes: geometric properties of the embedding space, and the narrowness of the text used to check the interpretation in the first place. Their stated conclusion is a direct warning against exactly the shortcut confirmation bias takes — activations of individual neurons “may spuriously appear to encode a single, simple concept” — and an interpretation validated on one dataset is, on their evidence, not validated at all [2].
The corrective the field has actually built is a set of criteria that force a circuit claim to survive contact with cases it was not written to explain. The indirect-object-identification circuit in GPT-2 small is the clearest instance: twenty-six attention heads sorted into seven functional classes, evaluated against three explicit tests its authors defined in advance — faithfulness, completeness, and minimality — rather than judged by whether the story sounded right [3]. Before criteria like that existed, “we found a circuit” had no defined way to fail, which is another way of saying it was not yet a claim that could be wrong.
A dictionary that reconstructs is not a dictionary that explains
Sparse dictionary learning promises a fix for polysemantic neurons: train an autoencoder to reconstruct a model’s activations through a sparse, overcomplete code, and read the code’s dictionary elements as candidate features. The objective it optimizes, though, refers only to the activation vector being reconstructed — it says nothing about what the model’s own downstream layers do with that code afterward. Low reconstruction error at high sparsity is evidence that the activation distribution is sparsely decomposable in the trained basis. It is not evidence that the basis is the one the network itself uses.
Two recent results make that gap concrete rather than philosophical. David Chanin and colleagues identified “feature absorption”: as a sparse autoencoder is pushed to split a hierarchical concept into finer sub-features, a parent feature that should fire on every member of a category instead fails to fire on some of them, because the signal has been silently absorbed into a more specific child feature that fires in its place. Using a task with fully known ground truth — first-letter identification, where every correct answer is checkable by hand — they showed the failure is not fixed by scaling the dictionary or tuning its sparsity; it is a structural consequence of what the sparsity objective rewards [4]. Separately, Subhash Kantamneni and colleagues ran close to a decisive downstream test: across four regimes chosen specifically to favor a genuinely useful feature basis — data scarcity, class imbalance, label noise, and covariate shift — they report they were “unable to design ensemble methods combining SAEs with baselines that consistently outperform ensemble methods solely using baselines” [5]. A dictionary can reconstruct its input beautifully and still not be the decomposition that makes anything else easier to do.
The picture that convinces nobody who tests it
Feature-visualization images — synthetic inputs optimized to maximally activate one unit — are among the most persuasive artifacts interpretability produces, because a reader can look at a grid of them and feel they understand what a component detects. The persuasiveness and the evidentiary weight are not the same thing, and the difference only shows up once the images are tested on people who did not select them for the paper. Papers rarely report the full distribution a feature-visualization method produces; they report the handful of crops that made the point cleanest, a selection procedure indistinguishable from the outside from cherry-picking, whether or not any author intended it that way.
Roland Zimmermann and colleagues ran the missing test directly: they asked both crowdworkers and domain experts to use feature-visualization images to predict which of two image regions would cause a larger change in a unit’s activation — a genuine causal-understanding task, not a subjective rating. The synthetic visualizations from the original 2017 feature-visualization method raised accuracy only modestly above having no visualization at all, and provided no meaningful advantage over the far simpler alternative of showing real dataset examples that activate the unit strongly [6]. That is an uncomfortable result for a method whose main appeal is intuitive persuasiveness: the images that look most self-evidently interpretable in a paper’s figure grid were, under a systematic test with a real behavioral criterion, barely better than a lookup table of ordinary examples.
What holds on a small model is a hypothesis about a large one, not a finding
The field’s most thoroughly triangulated results — the indirect-object-identification circuit, induction heads, the mathematical framework for attention-only transformers — were established on small models: GPT-2 small, or toy transformers built specifically to keep ground truth checkable. That was the right place to start, and it is also the reason the results do not automatically say anything about a frontier system. A mechanism demonstrated where it can be fully verified is an existence proof for models built like that one, not a measurement of models built at a different scale, trained on different data, and shaped by different post-training.
The most ambitious attempt yet to close that gap is itself the clearest evidence of how large it remains. Anthropic’s circuit-tracing method builds a “replacement model” out of interpretable features and traces attribution graphs through it — and its own authors report the replacement model matches the original model’s next-token output on roughly half of a diverse set of pretraining-style prompts, list attention-pattern formation as entirely outside what the method explains, and note that reconstruction error leaves a portion of the computation unaccounted for [8]. The companion paper applying the method to real behavior in Claude 3.5 Haiku — notably a lightweight production model, not the company’s largest — is candid about what its worked examples can and cannot support: the case studies are “existence proofs” that a mechanism can occur, the authors write, and they “cannot guarantee” similar mechanisms operate outside the examples shown, because what they found is itself “a biased sample shaped by the limitations of our tools” [7]. If the field’s most resourced circuit-tracing effort, applied to a smaller production model, produces findings its own authors describe that cautiously, a result obtained on a toy model several orders of magnitude smaller inherits at least as much uncertainty when someone asks whether it holds at frontier scale.
In enough dimensions, something always looks like a direction
Modern residual streams are thousands of dimensions wide, and a search for “the direction that correlates with concept X” is, formally, a search over an enormous number of candidates for one that clears some threshold. That structure alone should raise a familiar statistical concern: if enough candidates are tried, some will clear the threshold by chance, whether or not any of them is doing anything the model itself relies on.
Put simply: if a search evaluates
a quantity that climbs toward near-certainty long before
This is not a hypothetical worry about interpretability’s search procedures; a recent formal treatment shows the underlying scores are unstable enough that the concern is closer to guaranteed than merely possible. Maxime Méloux and colleagues reframed circuit discovery explicitly as a statistical estimation problem built on causal mediation analysis, and found that exact, single-input causal-mediation scores carry high intrinsic variance that propagates and compounds through a typical circuit-discovery pipeline: fast approximation methods add further noise, and aggregating scores across a dataset produces structural conclusions the authors describe as fragile, such that “minor shifts in input data or hyperparameters generate substantially different circuits” [9]. A circuit that looks like a clean discovery from one run of one pipeline may be one of several similarly plausible circuits the same noisy procedure would have returned on a slightly different day.
A metric that moves is not proof the mechanism you named moved it
Causal intervention is the right standard for a mechanistic claim — change a proposed component and see whether behavior changes as predicted, rather than merely noting an activation that correlates with an outcome. But intervention does not certify itself, and one of the field’s most important negative results shows exactly how a successful-looking intervention can support the wrong explanation.
Aleksandar Makelov, Georg Lange, and Neel Nanda studied subspace activation patching, in which a researcher identifies a low-dimensional subspace, patches it between a clean and a corrupted run, and treats the resulting behavioral change as evidence that the subspace carries the relevant feature. They showed the same behavioral change the technique is built to detect can instead be produced by activating a dormant, functionally unrelated pathway that happens to be reachable through the patched subspace — a pathway with nothing to do with the model’s ordinary computation on that input. Crucially, they did not construct this as an artificial worst case: they demonstrated it on two real tasks, indirect object identification and factual recall, the same tasks used to establish some of the field’s best-supported circuit claims [10]. The behavioral metric moved exactly as the hypothesis predicted, and the mechanism the paper’s own authors would have reported, absent the follow-up check, would have been wrong. Behavioral change under intervention is necessary evidence for a mechanistic claim. On this paper’s own evidence, it is not sufficient, and a result section that stops at “the metric moved” has not yet ruled out the dormant-pathway explanation for why it moved.
The check nobody ran before the paper went out
Peer review for an interpretability paper typically checks that the writing is clear, the method is described, and the headline claim is plausible given the reported numbers. It does not typically re-run the code against the released data, check whether the reported figures are reproducible from what was actually shared, or adversarially search for alternative explanations the authors did not think to rule out. That gap between what review checks and what would actually validate a claim is not specific to interpretability — Sayash Kapoor and Arvind Narayanan’s survey of data leakage across seventeen scientific fields that adopted machine-learning methods found the same shape of problem repeatedly: reported results that looked strong under the review the papers received, and considerably weaker once an independent team re-examined the pipeline for leakage between training and evaluation data [16].
A study built specifically for interpretability work shows the gap is not closing on its own. Xiaoyan Bai and colleagues built an automated evaluation framework that examines a paper’s code and data directly rather than only its prose, and applied it to published mechanistic interpretability research. Their system reached above 80 percent agreement with human expert judges on the problems both identified — and then surfaced 51 additional methodological issues that the human reviewers had missed entirely, catching problems the authors describe as often invisible without actually executing the code: missing data-selection criteria, validation logic that checks the wrong thing, files needed to reproduce a key result that are absent from the release [11]. The honest reading of that number is not that mechanistic interpretability researchers are unusually careless. It is that the standard review process was never built to catch this class of problem, in this field or in most others, and a check nobody runs is a check a paper can fail silently, for years, without anyone finding out.
The probe that passes because of what it was never asked
A probing study trains a simple classifier on a model’s internal representations to predict some property — part of speech, a factual attribute, a behavioral tendency — and reports the classifier’s accuracy as evidence the property is represented inside the model. The inferential step from “the probe scored well” to “the representation encodes this” quietly assumes something the accuracy score alone cannot establish: that a probe could not have scored just as well on a representation that did not encode the property at all.
John Hewitt and Percy Liang showed that assumption fails in exactly the cases that matter. They introduced control tasks — variants of a probing task where labels are assigned in a way a genuinely structured representation should not help predict — and found that some probing setups previously reported as evidence of rich linguistic structure scored well on control tasks too, meaning the probe itself, not the representation, was doing much of the work of memorizing an arbitrary mapping. Accuracy alone, without a control task establishing what a probe could achieve on a representation stripped of the property, does not distinguish a genuine finding from a powerful classifier fitting the specific dataset it was handed [12]. Yonatan Belinkov’s subsequent survey of the probing-classifier literature systematizes this and related concerns into a standing methodological caution for the field: probing results depend heavily on probe complexity, on the specific dataset and task used to elicit them, and on design choices frequently under-reported relative to the confidence with which the headline finding is stated [13]. A probing result computed on one dataset, with one probe architecture and no control task, is a data point about that dataset and that probe before it is evidence about the model.
Naming the wiring is not the same as trusting the switch
A circuit description and a control lever answer different questions, and a result that answers one is routinely read as though it answered the other. Knowing which components carry a signal is a claim about description. Being able to reliably change the model’s behavior by acting on those components is a claim about control. A model that reliably behaves as intended when a human is watching the output is a third, separate claim about actionability in practice — and recent evidence suggests near-perfect success at the first can coexist with a sizeable failure at the third.
A 2026 study of clinical-triage tasks put a sharp number on that gap. Sanjay Basu and colleagues found that linear probes trained on a model’s internal activations could distinguish hazardous from benign clinical cases with 98.2 percent AUROC — near-perfect internal detection, by the field’s usual standard for what counts as “the model knows this.” The model’s actual output-level behavioral sensitivity on that same distinction, measured by whether its final answers reflected the hazard, was only 45.1 percent, a fifty-three-point gap between what the internals encoded and what the model’s answers did with it. Attempting to close that gap by intervening directly on the internals did not simply work: concept-bottleneck steering corrected 20 percent of the missed hazards, but in the same experiment disrupted 53 percent of the cases the unmodified model had been getting right [14]. Describing where a piece of knowledge lives inside a network does not, by itself, tell you how to safely act on that description, and an intervention built directly from an accurate description can still make a system’s real behavior worse rather than better.
What happens when somebody else tries it
The test a published circuit claim least often receives is the most basic one available in any experimental science: an independent team, working from the paper and its released materials, tries to get the same result a second time. Independent replication is rare in mechanistic interpretability for an ordinary reason — it is unglamorous, unrewarded by the incentives that govern publication, and, per the variance result discussed earlier, liable to produce a different-looking circuit even when nothing was done wrong, simply because the underlying estimation procedure is noisy enough that a second run is not guaranteed to land on the first run’s answer [9].
What replication attempts do exist raise a further, more specific worry: an apparent replication is not always an independent one. Tal Haklay and colleagues, building and testing an autonomous system for interpretability circuit analysis, found that when large language models are used to reproduce or extend published interpretability findings, the systems can succeed for the wrong reason — by reproducing a published result “via memorization or informed guessing” about what the answer is expected to be, rather than by genuinely re-deriving it from the underlying model [15]. That is a caution about automated replication specifically, and it generalizes uncomfortably to human replication too: a researcher who already knows the published circuit before attempting to confirm it is not fully independent of that knowledge, and a field with few genuinely blind replication attempts has few results tested by someone who did not already know the intended answer. Until independent, ideally blinded reproduction becomes routine rather than exceptional, the honest status of most published circuits is “reported once, by the team with every incentive to find it” — a real result, and a weaker one than the paper’s confidence usually implies.
Predictions, with what would falsify them
These are forecasts, not findings, and are kept separate from the sourced analysis above. Horizon: 12 August 2029. Assumptions: mechanistic interpretability continues to be funded and published at roughly its current rate by frontier labs and academic groups, and no single automated method fully displaces manual circuit analysis in the interim.
One. Execution-grounded review — automated checking of a paper’s actual code and data rather than its prose alone — will become a standard part of at least one major interpretability venue’s review process, following the pattern the peer-review-gap findings above suggest is needed. Disconfirmed if the leading interpretability venues in 2029 still rely solely on manuscript-only review with no code-execution or data-reproduction component.
Two. Reported feature and circuit claims will increasingly be accompanied by an explicit measure of estimation stability — a variance, a resampling check, or an equivalent — rather than a single point estimate from one run, as the statistical-estimation framing of circuit discovery gets absorbed into standard practice. Disconfirmed if the median published circuit-discovery paper in 2029 still reports results from a single run with no stability or resampling check.
Three. The count of published interpretability claims subjected to a genuinely independent, blinded replication attempt will remain small relative to the total number of claims published, because the incentive structure that makes replication unrewarded is a funding and career problem, not a technical one, and better tooling does not by itself fix it. Disconfirmed if blinded replication becomes a standard requirement before a circuit or feature claim is treated as established, rather than the exception it is today.
Four. At least one further published, well-cited circuit or feature interpretation will be shown to be substantially wrong or incomplete by an independent audit, joining the documented cases already on record above. Disconfirmed if the 2026–2029 period passes with no such correction to a prominent published claim.
What to take away
None of the ten failure modes above requires an author acting in bad faith, and none is a reason to distrust interpretability research wholesale — several of the studies cited here are themselves interpretability research policing its own methods, which is a healthier sign than a field with no such studies at all. What they share is a single structural cause: a trained network offers an enormous space of superficially plausible stories about what it is doing, and only a small number of checks — a control task, an adversarial rerun, a held-out replication, a systematic test against people who were not the authors — actually narrow that space down to the story that is true. Skipping any one of those checks does not make a result false. It makes the result’s truth unverified, which a reader cannot tell apart from a result’s truth by reading the paper alone.
The practical rule for anyone using a published circuit or feature claim is the same rule this article’s own sourcing follows: ask what specific check was run, not merely what claim was made, and treat a claim with no documented adversarial check the way an editor should treat a stamp not yet on the page — closer to a decision made than a decision earned.
