What does this equation mean?

Anthropic’s own engineering guidance, published in September 2025 as advice for developers building long-running agents, gives the clearest available first-party account of why context rot happens architecturally rather than treating it as an unexplained empirical curiosity [ 4 ] . The explanation rests on the transformer’s core mechanism: every token attends to every other token in the context through self-attention, so the number of pairwise relationships the model must represent grows with the square of the sequence length. For a context of n tokens, the compute spent by self-attention within a single layer scales as $C_{\mathrm{attn}}(n) = O(n^2 \cdot d)$. where d is the model’s hidden dimension. Doubling the context does not double the work of relating everything in it to everything else; it roughly quadruples that specific cost. Anthropic’s guidance frames the practical consequence in the language of a finite resource: models have what it describes as a limited attention budget, and every additional token spends a sliver of it, so a model’s capacity to represent all the pairwise relationships in a very long input “gets stretched thin” well before the window is technically full [ 4 ] . The guidance names two further contributing factors distinct from the raw attention-cost argument: models see comparatively few long sequences during training relative to short ones, leaving fewer specialised parameters for context-wide dependencies, and the position-encoding schemes that let a model handle sequences longer than anything seen in training necessarily reduce the model’s precision about exactly where in a long sequence a given token sits [ 4 ] . Put together, Anthropic characterises context rot as “a performance gradient rather than a hard cliff” — a real, current-generation Claude behaviour, described in its own engineering team’s words, not a defect specific to a competitor’s system, and not a problem a strictly bigger window automatically fixes [ 4 ] .