Symbol Δ

The passage around this formula

Once the question is scoped, the experiment that tests it is a causal one by convention in this field, and the convention exists for a good reason: a component correlated with a behaviour has told you nothing about whether the model uses it, while a component that changes the behaviour when intervened on has told you something. The standard instrument is activation patching — running the model on a clean input, running it again on a corrupted one, then substituting the corrupted run’s activations into the clean run at a chosen set of components $\mathcal{C}$ and measuring the change in some behavioural metric m : $\Delta(\mathcal{C}) = m\big(M_{\mathrm{clean} \to \mathcal{C} \leftarrow \mathrm{corrupt}}\big) - m\big(M_{\mathrm{clean}}\big)$. The choice buried inside that formula — what counts as…

Read this part in the article →