$$\mathrm{SPEM}_{\mathrm{softmax}}$$

Equation 20 · Language Models & Evaluation

Cognitive Bias Was the Label; Attention Sink Is the Suspect

This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.

where $\mathrm{SPEM}_{\mathrm{softmax}}$ is Guo and Vosoughi’s own Jensen–Shannon-divergence effect magnitude, measured by their own published protocol, on an ordinary sink-forming model, and $\mathrm{SPEM}_{\mathrm{sigmoid}}$ is the identical measurement on an architecture- and data-matched twin trained with sigmoid attention instead of softmax — the exact contrast Gu and colleagues’ released training pipeline was built to produce [ 3 ] . The quantity is dimensionless, bounded in the neighborhood of zero to one for any model pair where the sink-free twin is not somehow more position-sensitive than its sink-forming counterpart, and it answers a single, sharp question: of the effect magnitude the…

Equation guide → · Article →