Symbol q

The passage around this formula

The property that makes this useful rather than merely elegant is that rotating a query at position m and a key at position n by these matrices before taking their dot product leaves a result that depends only on the distance between them: $\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle$. Relative position falls out of the mechanism instead of being engineered into it separately, and — as the RoFormer paper’s own experiments emphasise — the same rotation formula applies at any sequence length the rotation frequencies were designed for, which is what makes RoPE the natural place to intervene when a later generation wants a longer context window [ 5 ] .

Read this part in the article →