Equation 12 · Part 1 · How Llama's Architecture Actually Works, Generation by Generation
Symbol d
What this part means
d is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Its job in the formula
d is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Full expression→Symbol d→Article meaning
The passage around this formula
The one free parameter is base , the constant that sets how quickly the rotation angles grow across the d/2 frequency bands. A small base makes low-index pairs rotate almost a full turn per token, encoding fine-grained short-range position; a large base slows that rotation down, which is what lets high-index pairs still carry usable positional signal at sequence positions far beyond what the model saw in training. Llama 1 and Llama 2 shipped with the RoPE default of base = 10,000, matching the value in Su and colleagues’ original formulation. Llama 3 raised it to 500,000 specifically as part of extending context to 128,000 tokens [ 5 , 2 ] . Nothing about the rotation mechanism changed…
Learn the underlying idea
A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.
Open the illustrated variables: a letter stands for a value guide →
See this notation across published equations →
Sources cited in the article section
These citations provide research context; check each source for the exact claim it supports.