← All parts of this equation

Equation 19 · Part 16 · How Llama's Architecture Actually Works, Generation by Generation

Numerator: 1

RMSNorm(x)j=xjRMS(x) gj,RMS(x)=1d∑k=1dxk2+ϵ.\mathrm{RMSNorm}(x)_j = \frac{x_j}{\mathrm{RMS}(x)}\, g_j, \qquad \mathrm{RMS}(x) = \sqrt{\frac{1}{d}\sum_{k=1}^{d} x_k^2 + \epsilon}.
11

What this part means

The complete quantity above the fraction bar.

Its job in the formula

1 occurs above the fraction bar. The numerator is divided by the entire denominator below it.

The passage around this formula

…. Llama 2’s architecture section lists RMSNorm as the pre-normalization scheme applied before each attention and feed-forward sublayer, and it has not been revisited in any subsequent generation’s public documentation [ 1 ] .

Read this part in the article →

Learn the underlying idea

A fraction a/b means a divided by b. The top number is the numerator; the bottom number is the denominator, and it cannot be zero.

Open the illustrated fractions: division written vertically guide →

Sources cited in the surrounding passage

These citations provide research context; check each source for the exact claim it supports.