← All parts of this equation

Equation 11 · Part 6 · How Llama's Architecture Actually Works, Generation by Generation

=

⟨RΘ,m q,  RΘ,n k⟩  =  ⟨RΘ, m−n q,  k⟩.\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle.
=

What this part means

The expressions on both sides represent the same quantity under the stated assumptions.

Its job in the formula

The equals sign connects the complete expression on the left with the complete expression on the right. Both sides must have compatible units.

The passage around this formula

The property that makes this useful rather than merely elegant is that rotating a query at position m and a key at position n by these matrices before taking their dot product leaves a result that depends only on the distance between them: ⟨RΘ,m q,  RΘ,n k⟩  =  ⟨RΘ, m−n q,  k⟩\langle R_{\Theta,m}\,q,\; R_{\Theta,n}\,k\rangle \;=\; \langle R_{\Theta,\,m-n}\,q,\; k\rangle. Relative position falls out of the mechanism instead of being engineered into it separately, and — as the RoFormer paper’s own experiments emphasise — the same rotation formula applies at any sequence length the rotation frequencies were designed for, which is what makes RoPE the natural place to intervene when a later generation wants a longer context window [ 5 ] .

Read this part in the article →

Learn the underlying idea

An equals sign says that the expression on its left and the expression on its right have the same value under the stated definitions and assumptions.

Open the illustrated equality: what the equals sign claims guide →

Sources cited in the surrounding passage

These citations provide research context; check each source for the exact claim it supports.