The seal, and what it is supposed to protect
A previous article in this series worked through what a mechanistic account of a neural network is actually claiming, and set out the field’s own standard for taking such a claim seriously: change the thing you think matters and see whether the model’s behaviour changes with it, rather than merely noticing that the thing correlates with the behaviour. That standard is now well established in the literature. It is also, on its own, incomplete, and the incompleteness is not a footnote — it is the reason a reader outside the field cannot yet tell a rigorously supported interpretability result from a plausible-sounding one just by checking whether the word “causal” appears in the abstract.
This article asks a narrower and more practical question: once a paper says it intervened, how do you know the intervention itself was set up correctly, checked against anything, or repeated by anyone who was not already invested in the answer? The machinery the field has built to answer that question has three parts. The first is a way of telling a description from a check. The second is the deliberate construction of small systems whose true mechanism is known in advance, so a method can be graded against an answer key before it is trusted on a system nobody has solved. The third is independent replication, which turns out to be the least common of the three, and whose rare appearances are correspondingly the most informative. Two documented cases below show what happens when it does appear: a published claim gets revised, and a widely cited localization result turns out not to predict the thing everyone assumed it predicted.
A feature visualization is a description, not a check
Start with the failure mode that makes the rest of this necessary. Feature visualization — synthesizing an input that maximally activates a unit, then reading the resulting image as “what that unit represents” — was the founding technique of the circuits programme and remains one of the most persuasive-looking artifacts interpretability produces. It is also, on the best available evidence, not a check on anything by default.
Geirhos and colleagues constructed networks containing a deliberately hidden circuit and showed that standard feature-visualization methods can be made to display an arbitrary pattern of the attacker’s choosing, one with no connection to how the network actually processes natural images [1]. That alone might be dismissed as an adversarial curiosity confined to specially built networks. Their second result is harder to set aside: even in ordinary, unmanipulated networks, the optimized images that feature visualization produces are processed by later layers in a measurably different way than natural inputs are, which means the visualization is not obviously a window onto the network’s normal behaviour at all. They go on to prove, for a defined class of networks, that the set of functions for which feature visualization is provably reliable is small and does not include the black-box networks the technique is routinely applied to in practice.
None of this means feature visualization is useless — it remains a source of hypotheses worth checking. It does mean that a figure showing “what a unit looks for,” unaccompanied by any test of what happens when that unit is altered, is closer to a plausible narrative than to evidence. The distinction matters because narrative and verification produce output that looks identical on the page: a picture, a caption, and an air of having explained something.
What a check actually requires: an answer you already have
Causal intervention is the field’s answer to the visualization problem, and it is a real improvement — patching a component and watching a behavioural metric move is evidence of a kind that a picture of an activation maximum simply is not. But intervention creates its own version of the same problem one level up. A patching experiment produces a number: how much did the metric change when this component was clamped to a different value. Whether that number should be read as “this component causes the behaviour” depends on choices — which corrupted input, which metric, which direction of the patch — that are not fixed by the method itself, and different reasonable choices have been shown to yield different conclusions about the same behaviour in the same model.
So the question recurses: how do you know your causal-intervention protocol is sound? The only fully satisfying answer is to run it on a system where the true mechanism is already known by construction, and check whether the protocol recovers that known answer. A method that cannot rediscover a mechanism nobody is in doubt about should not be trusted to discover one that is actually in question. This is the precise, narrow job of what the field calls a model organism.
Model organisms: building the answer key first
A model organism, in this sense, is a network deliberately built or selected so that its ground-truth algorithm is known before any interpretability method is pointed at it — the interpretability equivalent of validating a diagnostic test against a patient whose condition is already confirmed by other means, rather than trusting the test’s first real-world application.
The cleanest version compiles the answer key directly into the weights. Lindner and colleagues built Tracr, a compiler that turns a human-written program into transformer weights that implement it exactly, so the resulting network’s algorithm is not inferred after the fact but specified before training even happens [2]. A circuit-discovery method run on a Tracr-compiled network can be graded against ground truth the way a spell-checker can be graded against a dictionary. The limitation is equally direct: compiled networks are cleaner and more modular than anything gradient descent produces, so passing this test is necessary and clearly not sufficient. Gupta and colleagues built InterpBench specifically to close that gap, training semi-synthetic transformers with a stricter interchange-intervention procedure that keeps a known circuit intact while giving the rest of the network realistic, gradient-descent-like structure around it, and extending the approach to circuits as large as indirect object identification rather than only small hand-compiled programs [3].
A second kind of model organism is not compiled but grown, then fully solved by hand. Nanda and colleagues trained small transformers to perform modular addition and reverse-engineered the resulting network completely: it computes the answer using a discrete Fourier transform and trigonometric identities that convert addition into rotation around a circle, and the training dynamics show three distinct phases — memorization, gradual circuit formation, and a later cleanup phase that removes the memorized shortcuts [4]. This is a genuinely trained network, with a genuinely surprising algorithm nobody designed by hand, but it is small enough that the account could be checked exhaustively rather than sampled. It functions as a model organism retroactively: once solved, it becomes a reference other circuit-discovery methods can be run against and graded on.
The common thread is that a model organism validates a method, not a claim about any particular frontier system. Passing the test tells you the method can, in this controlled setting, find what is actually there. It does not tell you what your method found when you later ran it on a production model, because a production model was never solved independently — that is the entire reason you needed the method.
The floor is lower than it looks: benchmarking against a known bug
A related but distinct validation move implants a specific, known flaw into an otherwise realistic network and asks whether interpretability tools can find it, which tests usefulness on something closer to an engineering task than a scientific one. Casper and colleagues trained networks containing twelve deliberately implanted “trojans” — inputs that trigger a specific unintended behaviour — spanning three trigger types, then evaluated sixteen feature-attribution methods and seven feature-synthesis methods on whether they could help a human locate the trojan without already knowing the trigger [5]. Their finding is sobering rather than encouraging: most of the tested methods regularly failed to identify the planted bug even with privileged access to data containing the trigger, which is a considerably easier setting than the real one, where nobody hands you the input that breaks the model.
This kind of benchmark descends from an older and more basic validation move: check whether an explanation changes when it should have to. Adebayo and colleagues showed that several popular saliency methods for image classifiers produce visually similar output whether the underlying model has been trained at all or left at its random initialization, meaning the “explanation” was substantially independent of what training had actually done [6]. Stated as a minimal necessary condition, if
An explanation that is identical whether or not training happened cannot be reporting anything about what training did; it is a function of the architecture and the input alone, dressed up as a function of the model. This is a floor, not a ceiling — passing it establishes only that a method is not obviously vacuous, and it is the kind of check that a strikingly large share of published interpretability results have never been run against.
Independent replication, and its frequent absence
Model organisms and implanted-bug benchmarks are both, in effect, a research group checking its own method against a known answer before applying it elsewhere. The remaining and rarer form of validation is a different group checking someone else’s finding, ideally without a stake in the original result coming out right. It is rare for an ordinary reason: reproducing an interpretability result usually means re-implementing a pipeline, re-running expensive interventions, and — crucially — testing a prediction the original authors did not think to test, which is more work than citing the original claim and moving on.
One case where that extra work was done concerns causal tracing, the technique behind one of the field’s most cited localization results. Meng and colleagues used causal tracing to argue that factual associations in GPT-style models are concentrated in specific middle-layer feed-forward modules at the subject token, and used that localization to guide a targeted weight edit, Rank-One Model Editing, that successfully inserted new facts [8]. The natural reading of that result is that the localization is doing real explanatory work: find where a fact lives, then edit it there. Hase and colleagues tested that reading directly, checking whether the layer causal tracing identifies as most responsible for a fact is in fact the best layer to edit when overriding, erasing, or amplifying it. It is not. Across their experiments, which layer produced the best editing outcome was far better predicted by properties of the editing method itself than by the causal-tracing localization, and successful edits could be made at layers causal tracing had not flagged as important at all [7]. This is not a claim that ROME’s editing technique fails — it demonstrably works — but it is a documented case where an independent test showed that a widely repeated causal story about why it works does not hold up as stated. The localization and the editing success are both real; the inference from one to the other is the part that did not survive contact with a direct check.
A claim that was revised: the board that turned out to be linear
A second documented case is less a refutation than a correction, and it is worth walking through in detail because the correction is more informative than either version of the claim alone. Li and colleagues trained a GPT-style model purely to predict legal moves in the board game Othello, with no built-in knowledge of the game’s rules, and found what they described as an emergent nonlinear internal representation of the board state: nonlinear probes could decode the board from the model’s activations, and intervening on that nonlinear representation let them steer the model’s predicted moves [9]. The paper became one of the most cited pieces of evidence that sequence models trained only to predict the next token can build something resembling an internal world model, and the nonlinearity of the representation was treated as part of the finding.
Nanda, Lee, and Wattenberg revisited the identical model and task and found a linear representation instead, using a different framing of the probe: rather than asking whether a square holds black or white, they asked whether it holds “my colour” or “the opponent’s colour,” a distinction that flips at every move and that a fixed linear probe cannot see under the original framing but can see immediately under the revised one [10]. The board state, it turned out, was linearly decodable all along; the earlier nonlinear result reflected a choice of probing basis rather than a fact about the network’s geometry. This mattered beyond tidiness, because the linear representation hypothesis is the assumption nearly all subsequent interpretability tooling — probing, steering vectors, sparse dictionaries — is built on, and the corrected result is far more consistent with that broader picture than the original was. The self-correction worked exactly as intended, through an independent team re-running the same experiment with a different, better-motivated framing rather than through any retraction. It also took roughly a year to arrive, and it is considerably less well known than the finding it revised, which is itself evidence about how asymmetric attention is between an original claim and its correction.
When independent checking is designed in from the start
The strongest evidentiary design the field has produced so far builds the independent check into the experiment before anyone has a result to defend. Marks and colleagues, at Anthropic, trained a language model with a deliberately implanted hidden objective — first teaching it about a set of exploitable errors in reward models used for RLHF, then training it to exploit those errors while it learned to conceal that behaviour from evaluators — and then set four separate research teams, blind to the model’s hidden objective and to each other’s work, to investigate it for hidden problems [11]. Three of the four teams correctly identified the objective, using a mixture of techniques that included sparse-autoencoder-based interpretability, deliberately adversarial behavioural probing, and direct analysis of training data. A follow-up, unblinded phase then evaluated eight distinct auditing techniques individually for their strengths and blind spots.
This is the vendor’s own claim about its own model, and it should be read that way rather than as an independent audit of the field’s methods in general: Anthropic designed the ground truth, ran the game, and reported the results. Taken on those terms, it is still an unusually well-constructed piece of evidence, because a designed hidden objective, teams blind to it and to one another, and an explicit pass or fail outcome together resemble a controlled trial far more than the field’s default mode of one team publishing a finding that other teams then cite rather than test. It is also a limited result: interpretability contributed to, but was not uniquely responsible for, the successful audits, since teams with direct access to training data succeeded by other means as well, and the exercise covered one model, one kind of hidden objective, and one implanting procedure. Generalizing from it to “interpretability can now audit hidden objectives” would overstate what a single, self-run exercise established.
An evidentiary ladder
Putting the pieces above in order gives a rough hierarchy of evidentiary strength for an interpretability claim, weakest to strongest. A feature visualization or a hand-picked example that “looks right” sits at the bottom; Geirhos and colleagues’ result means this should be read as a hypothesis, never a conclusion, on its own [1]. Above that sits correlational probing — a direction in activation space that correlates with a labelled concept — which is stronger but still silent on whether the network uses that direction for anything. Above that sits a causal intervention with a stated, defensible protocol: a real ablation or patch on the actual system, with a control condition and a named metric. Above that sits validation of the method itself on a model organism with a known ground-truth mechanism, before its output on the real system is trusted, following Tracr and InterpBench [2, 3]. At the top sits independent replication, ideally under blind or pre-registered conditions, of the kind the ROME localization claim received and did not fully survive, and of the kind the Othello representation claim received and was revised by [7, 10].
Most published mechanistic interpretability work currently sits at the third rung: a real causal intervention, on the real system, without independent replication and without having been checked against a model organism first. That is not a condemnation — it is a genuine improvement over pure visualization, and it is what a young field working on frontier-scale systems that nobody can fully solve by hand can realistically produce at volume. But it is worth naming plainly, and the field’s own most recent agenda-setting review does so: a coordinated group of researchers spanning several of the field’s leading labs and universities lists methodological rigor and validation, rather than any specific technique, among the central open problems facing the discipline [12]. Researchers disagree, reasonably, about how to respond to that gap. One position holds that scaling existing causal methods to frontier models now, while accepting known unfaithfulness, generates more useful knowledge than it costs. The opposing position holds that building out the model-organism and replication infrastructure first, even at the cost of slower progress on frontier claims, is what keeps the field’s conclusions trustworthy once they matter for deployment decisions. Both positions are held by people who have read the same papers cited here; this is not a dispute that more data alone will settle, because it is partly a dispute about which mistake is more costly to make.
Predictions, with what would falsify them
These are forecasts, kept explicitly separate from the evidence above. Horizon: 12 August 2029.
One. Model-organism benchmarks in the style of Tracr and InterpBench will become a standard reviewer expectation for new circuit-discovery methods, the way held-out test sets already are for supervised learning. Disconfirmed if a majority of accepted circuit-discovery papers at the field’s leading venues in 2029 still report results only on frontier-scale models with no synthetic or semi-synthetic validation step.
Two. Designed, blind auditing exercises modeled on the hidden-objective game will spread beyond the lab that ran the first one, run by parties without a stake in the outcome. Disconfirmed if by 2029 no interpretability-relevant blind auditing exercise has been conducted or reported by a team auditing a model it did not itself build.
Three. At least one more prominent, already-published interpretability claim will be materially revised by an independent replication attempt, following the pattern of the Othello and causal-tracing cases here. Disconfirmed if the field’s citation record through 2029 shows no comparable correction to a widely cited result.
Four. The gap between causal-intervention protocols will narrow as standardized, shared tooling emerges, reducing the degree to which the same patching experiment yields different conclusions depending on which lab ran it. Disconfirmed if published replications in 2029 still routinely report protocol-dependent disagreement about circuit membership for the same behaviour in the same model.
What to take away
A feature visualization is a hypothesis with a picture attached, not a finding. A causal intervention is stronger evidence, but its own protocol is a set of choices that can be checked only by testing it against something whose answer is already known — which is what a model organism is for, and what implanted-bug benchmarks test on a more practical, engineering register. Independent replication, the strongest and rarest rung, has already caught one localization claim that did not predict what it was assumed to predict, and revised another from nonlinear to linear without ever retracting the original paper. None of this means the field’s claims are unreliable; it means the claims that have actually been checked this way are the ones worth the most weight, and a reader should ask, of any interpretability result presented as settled, which of these checks it has actually been through.
