← All parts of this equation

Equation 11 · Part 3 · How Llama's Architecture Actually Works, Generation by Generation

Symbol R_Theta,n

⟨RΘ,m q,  RΘ,n k⟩  =  ⟨RΘ, m−n q,  k⟩.\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle.
RΘ,nR_{\Theta,n}

What this part means

RTR_Theta,n is part of the quantity the equation computes from the expression on the right.

Its job in the formula

RTR_Theta,n is part of the quantity the equation computes from the expression on the right.

The passage around this formula

The property that makes this useful rather than merely elegant is that rotating a query at position m and a key at position n by these matrices before taking their dot product leaves a result that depends only on the distance between them: ⟨RΘ,m q,  RΘ,n k⟩  =  ⟨RΘ, m−n q,  k⟩\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle. Relative position falls out of the mechanism instead of being engineered into it separately, and — as the RoFormer paper’s own experiments emphasise — the same rotation formula applies at any sequence length the rotation frequencies were designed for, which is what makes RoPE the natural place to intervene when a later generation wants a longer context window [ 5 ] .

Read this part in the article →

Learn the underlying idea

A subscript is a label attached below a symbol. It often selects a time step, component, category, or member of a sequence.

Open the illustrated subscripts: which member of a family? guide →

See this notation across published equations →

Sources cited in the surrounding passage

These citations provide research context; check each source for the exact claim it supports.