\operatorname{RMSNorm}

RMSNorm — root-mean-square normalization

Read "R M S norm". It is LayerNorm without the mean subtraction and without the shift (β): divide each vector by its root mean square, then apply a learned scale.

Root mean square means: square each entry, take the mean, then the square root. $\operatorname{RMS}(\mathbf{x}) = \sqrt{\frac{1}{d}\sum_i x_i^2 + \epsilon}$ , and the layer outputs $\frac{\mathbf{x}}{\operatorname{RMS}(\mathbf{x})} \odot \boldsymbol{\gamma}$ .

Skipping the mean saves one pass over the vector. In practice it works about as well as LayerNorm, which suggests the rescaling does the useful work, not the re-centring. Gemma, LLaMA and Qwen all use it.

Example

\operatorname{RMS}([3, 4]) = \sqrt{\tfrac{9 + 16}{2}} \approx 3.54 \;\Longrightarrow\; [0.85,\, 1.13]

Ignoring ε, with γ = 1. The mean is not subtracted.

Where the course uses it

15.3 RMSNorm · Module 15, The LLaMA Blockfirst use
16.4 Sandwich Normalization and Embedding Scaling · Module 16, Efficient and Long Attention

Related terms

LayerNorm — layer normalizationRead "layer norm". It shifts and rescales each token's vector to have mean 0 and variance 1, then applies a learned scale and shift.
ε — a tiny safety constant (or the CTC blank)Read "epsilon". It is a very small number added to a denominator so the division can never be by zero. In CTC (section 27.2) the same letter means something else: the blank symbol, "no character here".