What does this equation mean?

The first concrete operation in almost any interpretability project is the least glamorous: run the model forward on a batch of inputs, and at some chosen point in the computation — a residual-stream position, an attention head’s output, a particular MLP layer — copy the activation tensor out before it is overwritten by the next step of the forward pass. This is extraction, and it produces nothing on its own beyond a large table of vectors. What turns it into evidence is probing: fitting a small, separately trained classifier to predict some property of interest directly from those vectors, while the model’s own weights stay frozen. Formally, for an activation $a_\ell$ read out at layer $\ell$ and a binary property y , $\hat p_\theta(y \mid a_\ell) = \sigma\!\left(w^{\top} a_\ell + b\right), \qquad \theta = \{w, b\}$. with $\theta$ fit by ordinary gradient descent to minimise cross-entropy against labelled examples. Alain and Bengio introduced this move under the name “probes” and made an observation that still organises how the technique is used: linear separability of a target property increases monotonically with depth in the networks they studied, which is itself informative about where in the computation a property becomes available for linear readout [ 1 ] .