multiplication

The passage around this formula

Anthropic’s own engineering guidance, published in September 2025 as advice for developers building long-running agents, gives the clearest available first-party account of why context rot happens architecturally rather than treating it as an unexplained empirical curiosity [ 4 ] . The explanation rests on the transformer’s core mechanism: every token attends to every other token in the context through self-attention, so the number of pairwise relationships the model must represent grows with the square of the sequence length. For a context of n tokens, the compute spent by self-attention within a single layer scales as $C_{\mathrm{attn}}(n) = O(n^2 \cdot d)$. where d is the model’s hidden dimension. Doubling…

Read this part in the article →