← Mathematical compendium

Published equation contexts

⟨RΘ,m q,  RΘ,n k⟩  =  ⟨RΘ, m−n q,  k⟩\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle

Why this formula appears here

The property that makes this useful rather than merely elegant is that rotating a query at position m and a key at position n by these matrices before taking their dot product leaves a result that depends only on the distance between them: ⟨RΘ,m q,  RΘ,n k⟩  =  ⟨RΘ, m−n q,  k⟩\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle. Relative position falls out of the mechanism instead of being engineered into it separately, and — as the RoFormer paper’s own experiments emphasise — the same rotation formula applies at any sequence length the rotation frequencies were designed for, which is what makes RoPE the natural place to intervene when a later generation wants a longer context window [ 5 ] .

Read the full article-specific guide →

Read the representative guide

RΘ,mR_{\Theta,m}

Symbol R_Theta,m

RTR_Theta,m is part of the quantity the equation computes from the expression on the right.

Read this term in its guide →
RΘ,nR_{\Theta,n}

Symbol R_Theta,n

RTR_Theta,n is part of the quantity the equation computes from the expression on the right.

Read this term in its guide →
RΘ, m−nR_{\Theta,\,m-n}

Symbol R_Theta,m-n

RTR_Theta,m-n is one of the signed contributions combined to compute the quantity on the left.

Read this term in its guide →

How to interpret it

Read it with the definitions, units, and assumptions supplied by the article.

Research cited beside this formula

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

⟨RΘ,m q,  RΘ,n k⟩  =  ⟨RΘ, m−n q,  k⟩.\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle.

Equation 11 · Open Models

How Llama's Architecture Actually Works, Generation by Generation

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

The property that makes this useful rather than merely elegant is that rotating a query at position m and a key at position n by these matrices before taking their dot product leaves a result that depends only on the distance between them: ⟨RΘ,m q,  RΘ,n k⟩  =  ⟨RΘ, m−n q,  k⟩\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle. Relative position falls out of the mechanism instead of being engineered into it separately, and — as the RoFormer paper’s own experiments emphasise — the same rotation formula applies at any sequence length the rotation frequencies were designed for, which is what makes RoPE the natural place to intervene when a later generation wants a longer context window [ 5 ] .

Equation guide → · Article →