Denominator: RMS(x)

The passage around this formula

RMSNorm, from Zhang and Sennrich, made a narrower and more surgical change to normalization. Standard LayerNorm re-centres a layer’s inputs to zero mean and rescales to unit variance before applying a learned gain. Zhang and Sennrich’s hypothesis, tested empirically, was that the re-centring step is not doing useful work — that rescaling invariance alone accounts for LayerNorm’s benefit — and their RMSNorm drops the mean-subtraction entirely: $\mathrm{RMSNorm}(x)_j = \frac{x_j}{\mathrm{RMS}(x)}\, g_j, \qquad \mathrm{RMS}(x) = \sqrt{\frac{1}{d}\sum_{k=1}^{d} x_k^2 + \epsilon}$. Their reported result was performance comparable to LayerNorm at a running-time reduction of roughly seven to sixty-four percent depending on the model, purely from removing the mean and its gradient computation [ 7 ] . Llama 2’s…

Read this part in the article →