Running an Interpretability Investigation That Holds Up

An investigation, not a demonstration

A slide deck that says “we found the circuit for X” is a demonstration: it shows the audience a result. An investigation is a different object entirely, and most of the difference is invisible in the final write-up. It is the sequence of decisions made before the first attention pattern was ever plotted — what would count as being wrong, what has to be held still while something else changes, who is trying to break the claim before it leaves the building.

This matters more in mechanistic interpretability than in most adjacent fields because the object under study is unusually forgiving of sloppy method. A trained network is enormous, its activations are high-dimensional, and almost any intervention on almost any component will move almost any behavioural metric by some nonzero amount. That is not a sign you have found something. It is the background condition you are working against. A practitioner’s job is to design the investigation so that a nonzero effect on its own cannot be mistaken for a mechanism, and the tools for doing that are not exotic — they are the ordinary discipline of controls, held-out checks, and honest reporting, applied to a domain where they are still inconsistently practiced.

This is a practitioner’s guide to that discipline: how to scope a question so it can fail, how to build the causal experiment so a confound cannot pass for a finding, how to catch your own motivated reasoning before it reaches the page, how to write the result down without inflating it, and what happens to it once someone outside your project has to decide whether to rely on it.

Scoping the question so it can fail

The first practitioner decision, and the one skipped most often, is naming what kind of claim you are actually making before you make it. Doshi-Velez and Kim’s early framework for evaluating interpretability work distinguishes three grounds on which an interpretation can be judged: a functionally-grounded evaluation, which substitutes a formal proxy for human judgment because no human study is available; a human-grounded evaluation, which puts the interpretation in front of people doing a simplified task; and an application-grounded evaluation, which tests the interpretation inside the real downstream task it is meant to serve ^[1]. These are not interchangeable, and a result that clears one bar routinely gets reported as though it cleared all three. A high reconstruction score on a sparse dictionary is functionally grounded. It says nothing, by itself, about whether a human auditor reading the recovered features would catch a planted misbehaviour, and it says less still about whether the features hold up inside a live deployment. Before running anything, an investigator should be able to say in one sentence which of the three grounds the claim will be tested on, because that choice determines what evidence would even count against it.

The next decision is turning the claim into something with a stated failure condition. “This head does something related to indirect object identification” is not falsifiable; it cannot be wrong. The version of that claim the field actually treats as evidence looks like the standard set out by Wang and colleagues in their GPT-2 study of that exact circuit: a proposed set of components must satisfy faithfulness — the circuit alone reproduces the model’s behaviour when the rest is held fixed — completeness — no components outside the proposed set are doing real work — and minimality — no component inside the set can be removed without breaking it ^[2]. Each of those three criteria has an explicit pass/fail measurement attached before the circuit is presented as real, and each is a place the claim can die. Adopting that rubric, or one like it, before running the first patch is the interpretability-specific form of pre-registration: write down the metric, the input distribution the effect must hold across, and the size of effect that would count as a real finding, and do it before looking at the results, not after choosing whichever layer produced the cleanest plot.

The reason this ordering matters is not procedural fussiness. Zhang and Nanda’s systematic study of activation patching found that the choice of corruption distribution and evaluation metric — decisions usually made informally, late, and sometimes after a first look at the data — can by itself change which components a patching sweep identifies as important ^[3]. If the metric is chosen after the sweep, on the grounds that it produced the most legible result, the investigation has stopped testing a hypothesis and started constructing one to fit the data. The same failure has a clean statistical description. Sweep enough components at a nominal per-test false-positive rate $\alpha$ and report only the one that clears threshold, and the probability that at least one clears by chance alone across $k$ independently tested components is

which climbs toward certainty well before $k$ reaches the size of a typical layer-by-layer or head-by-head sweep. A held-out set of prompts, reserved and untouched until the hypothesis is fixed, is the practical fix, and it only works if the hypothesis really was fixed first.

The intervention experiment: what has to sit beside the change

Once the question is scoped, the experiment that tests it is a causal one by convention in this field, and the convention exists for a good reason: a component correlated with a behaviour has told you nothing about whether the model uses it, while a component that changes the behaviour when intervened on has told you something. The standard instrument is activation patching — running the model on a clean input, running it again on a corrupted one, then substituting the corrupted run’s activations into the clean run at a chosen set of components $\mathcal{C}$ and measuring the change in some behavioural metric $m$:

The choice buried inside that formula — what counts as “corrupt” — is not a detail. Zeroing an activation, replacing it with the dataset mean, or resampling it from an unrelated input each encode a different implicit null hypothesis about what the component would be doing if it were not doing the thing you suspect, and Zhang and Nanda’s comparison across these choices found they can support materially different conclusions about the same circuit in the same model ^[3]. An investigator has to pick one, justify the pick against what the hypothesis actually claims, and report which one was used; “we patched the component” without saying into what is not a complete method.

The more rigorous version of the same idea is causal scrubbing, developed at Redwood Research specifically to close a gap the field had been informal about: given a hypothesis that claims certain parts of a computation do not matter to a behaviour, resample precisely those parts — and only those parts — with activations from unrelated runs, while leaving everything the hypothesis calls load-bearing untouched, and check whether the behaviour survives ^[4]. A hypothesis that survives this treatment has been tested against its own claim of irrelevance, not merely shown to correlate with an outcome. A hypothesis that fails it has been falsified by the exact mechanism it proposed, which is a considerably stronger result than a patching sweep that only ever asks “does changing this move the metric.”

Neither method, on its own, rules out the most damaging kind of confound: an intervention that produces the expected behavioural change through a pathway that has nothing to do with the model’s normal computation. Makelov, Lange and Nanda demonstrated this concretely. Patching a subspace they had identified as carrying a feature produced the predicted behavioural shift — and the shift turned out to be driven by activating a dormant parallel pathway the network never uses under ordinary operation, not by the subspace doing what the hypothesis claimed, and they reproduced the effect on two real tasks rather than a constructed toy case ^[5]. The result the intervention produced was real. The explanation attached to it was not.

The defence against this is a control condition run alongside the target one, not after it: intervene identically on a component set $\mathcal{C}'$ matched to the target set $\mathcal{C}$ in size, depth and baseline activation statistics, but which the hypothesis explicitly predicts should have no effect, and compare

A specificity margin $S$ that is small relative to the run-to-run variance means the intervention is detecting generic sensitivity to perturbation somewhere in the network, not the mechanism under test — which is exactly the failure mode the subspace illusion exhibits. Reporting $\Delta(\mathcal{C})$ alone, without the matched control alongside it, is reporting half an experiment.

Automated circuit-discovery tools inherit the same discipline rather than replacing it. Conmy and colleagues’ ACDC prunes the computational graph by patching and was validated by checking that it recovered a previously hand-analysed circuit, all of it, using far fewer edges than the full graph contains ^[6]. That recovery is a legitimate check on the tool and a limited one: it shows the automated method does not miss what a careful human search already found in that one case, not that its output can be trusted un-audited on a case nobody has hand-verified. A tool that recovers known ground truth on a validation case still needs a control and a specificity check on the case that matters.

Where your own hypothesis will lie to you

None of the controls above defend against the investigator’s own motivated reasoning, which is the harder problem because it operates before any metric is computed — in which experiments get run, which get abandoned early, and which ambiguous result gets read charitably.

The clearest documented case of this correcting itself in public is Othello-GPT. Li and colleagues trained a transformer to predict legal moves in the board game Othello with no explicit rules given, then searched for an internal representation of the board state using nonlinear probes, and reported evidence of a nonlinear representation — a real, carefully-run result, published as an ICLR oral ^[7]. Nanda, Lee and Wattenberg went back to the same model and found a linear representation had been there the entire time, recoverable once the probe was framed around “my colour versus the opponent’s colour” rather than “black versus white” — the earlier framing had been aligned with the wrong symmetry of the task, and a negative search under it had been mistaken for a negative fact about the network ^[8]. Nobody in that sequence did careless work. The lesson is narrower and more useful than “be careful”: an absence of a signal is a fact about the search you ran, not a fact about the network, and it licenses only “not found by this method,” never “not present.”

The corresponding forward-direction failure — finding a signal that was never really there — has its own documented case. Heap and colleagues trained sparse autoencoders, the standard dictionary-learning tool for recovering candidate features, on transformers with randomly initialised weights, and found that automated interpretability scores and reconstruction metrics for the random networks came out comparable to scores for genuinely trained ones ^[9]. A metric that cannot tell a network that has learned language apart from one that has learned nothing is not a metric an investigator can trust to certify a specific finding as real, however clean the output looks. The practical guard this motivates is unglamorous and easy to skip under deadline pressure: run every interpretability metric on a matched random-weight control alongside the trained model, every time, and treat any result the metric cannot distinguish from its own random baseline as uninformative regardless of how legible it looks.

A related and older result names the generalisation failure specifically. Bolukbasi and colleagues showed that individual neurons and simple linear combinations of them in BERT can appear to encode one clean concept while actually encoding something considerably more tangled, and traced the illusion to two sources: geometric properties of the embedding space, and the narrowness of the corpus the interpretation happened to be checked against ^[10]. Their stated methodological conclusion is the operational fix: test the interpretation on more than one dataset before believing it, because an interpretation validated on a single corpus is, by their finding, not validated at all. In practice this means holding out an adversarially chosen or simply different dataset from the one used to form the hypothesis, and treating a hypothesis that only survives on the data that produced it as not yet a finding.

A structural guard worth adopting from outside interpretability altogether is adversarial replication inside the team: have one person or subgroup state the causal hypothesis and design the confirming experiment, and have a second, independent person or subgroup try specifically to construct the alternative explanation — a confound, a dormant pathway, a dataset artefact — that would produce the same observed effect without the proposed mechanism being true. This is the same structure, at team scale, that the alignment-auditing exercise discussed below uses at organisational scale: separating the people who know the answer from the people trying to find it is what makes the resulting confidence worth something ^[12].

Writing up a result you have not over-claimed

A patching experiment produces two different kinds of statement, and the write-up has to keep them visibly separate. The first is a measurement: intervening on component set $\mathcal{C}$ moved metric $m$ by some amount, with some variance across prompts and random seeds, and the matched control $\mathcal{C}'$ moved it by some smaller amount. That is data. The second is an interpretation layered on top of it: that the component “represents,” “detects,” or “computes” some human concept. The measurement is usually solid. The interpretation is usually the part doing more work than the evidence supports, and it is the part press coverage and even conference talks tend to keep while dropping every qualifier attached to it.

Concrete rules follow from the sections above and are worth stating as rules rather than aspirations. Name the corruption method and the metric explicitly — “we ablated the circuit” is not a method, “we mean-ablated components $\mathcal{C}$ and measured the change in logit difference on the IOI task” is. Report the specificity margin $S$ alongside the raw effect, not the raw effect alone. State the checkpoint identifier and the date it was evaluated, because the same architecture at a different training step is a different object. Say explicitly whether the claim has been tested outside the distribution it was found on, and if it has not, say that it has not rather than letting the reader assume otherwise.

The field’s better work already models this. Sharkey and colleagues’ agenda-setting review of open problems in the area is organised around what specifically remains unresolved rather than around a claimed inventory of what has been solved, which is itself a statement about how the leading practitioners think the evidence should be reported ^[11]. Casper and colleagues, arguing for stronger AI-audit access requirements, make a parallel point about disclosure as a norm rather than a courtesy: an audit’s credibility depends on stating exactly what access level and what methods produced a conclusion, because a black-box behavioural test and a white-box mechanistic one support very different strengths of claim about the same system, and conflating them in the write-up misleads the reader about how much was actually checked ^[13]. The discipline that section argues for at the level of an entire audit is the same discipline an individual investigation needs at the level of a single figure: say what was actually done, at what confidence, and stop there.

What an audit can actually use

The reason this level of care is not merely academic is that interpretability findings are already being fed into decisions about whether to trust a deployed model, and the field has direct evidence about how much weight they can currently bear.

The most informative test to date is a deliberately adversarial one. Marks and colleagues trained a language model with a concealed objective — exploiting known flaws in a reward model — and set four separate teams to find it under blind conditions, none of them told what they were looking for ^[12]. Three of the four teams succeeded, and they succeeded using different tools: interpretability analysis built on sparse autoencoders, behavioural red-teaming, and direct examination of training data. That result calibrates the claim correctly in both directions. Interpretability contributed materially to a real detection. It was not the only route that worked, and a team relying on it exclusively, without the other two strands, would have had a narrower chance of success than the paper as a whole demonstrates.

The case for why behavioural testing alone is not enough comes from a different experiment. Hubinger and colleagues trained models with a backdoored behaviour — writing safe code under one stated condition and inserting exploitable code under another — and found the backdoor survived supervised fine-tuning, reinforcement learning, and adversarial training aimed at removing it; in some configurations, adversarial training made the model better at concealing the behaviour rather than eliminating it ^[14]. A model that passes standard safety training with a concealed behaviour intact is, by construction, a model that a purely behavioural audit is not equipped to catch, which is the direct argument for why access to internals — not necessarily full interpretability, but at minimum white-box access of the kind Casper and colleagues argue is a precondition for a rigorous audit — needs to be part of the evaluation rather than an optional extra ^[13].

Bereska and Gavves’ review of the field from a safety standpoint is useful here for separating what interpretability is currently good for from what it is being asked to be good for. They organise the safety-relevant uses into distinct categories — explaining a behaviour after the fact, detecting a property before deployment, and controlling a behaviour once found — and the honest reading of the current evidence, consistent with everything above, is that these three uses are not equally mature ^[15]. Control is where interpretability is furthest along: a specific, reliably reproducible handle, once found and validated with the controls described earlier in this article, can be wired directly into a production safety stack without requiring a complete account of why the network is organised that way. Detection is closer behind, aided by exactly the kind of adversarial, blind auditing exercise Marks and colleagues ran. Certification — a positive claim that a property is absent across the full input distribution, not merely undetected on the cases tested — is the use case current methods are least equipped to support, because it demands coverage, and coverage is precisely what a handful of validated circuits and a partial feature dictionary do not provide.

The operational consequence for anyone scoping a real investigation is to match the ambition of the research question to which of these three uses it is actually meant to serve. An investigation aimed at building a control handle can succeed with a narrow, well-validated, matched-control finding of the kind described throughout this piece. An investigation aimed at detecting a specific, previously hypothesised concern can succeed with the blind, adversarial structure the auditing exercise used. An investigation that sets out to certify the general absence of a bad property is scoped, on the field’s own evidence, past what the tools in hand can currently deliver — and the honest report says that, rather than quietly narrowing the claim in the abstract while the introduction implies something broader.

A short protocol

Collected into one sequence, the practices above are: state which of the three evaluation grounds the claim will be tested on before running anything; write the metric, the corruption distribution and the predicted effect size down before looking at results, and hold out data the hypothesis was not shaped against; choose and justify the counterfactual an intervention implies, and run a matched control condition alongside the target one so a generic perturbation cannot be mistaken for a specific mechanism; test on more than one dataset, and against a random-weight baseline, before believing a result your own search produced; separate the measured effect from the interpretive claim built on top of it in the write-up, and state the checkpoint, the date, and what was not tested; and scope the investigation’s ambition — control, detection, or certification — to match what the current evidence says each actually supports.

None of this is exotic scientific method dressed up for a new field. It is closer to what a careful experimentalist in any causal discipline already does, applied to a substrate that is unusually easy to get a plausible-looking wrong answer from. The difference between a demonstration and an investigation that holds up is almost entirely made of these decisions, and almost none of them are visible in the figure that eventually gets shown.