LayerNorm — layer normalization

$\operatorname{LayerNorm}(\mathbf{x}) = \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} \odot \boldsymbol{\gamma} + \boldsymbol{\beta}$. Here $\mu$ and $\sigma^2$ are the mean and variance of the numbers within one token's vector. So the result for one token does not depend on the other examples in the batch, unlike BatchNorm. $\boldsymbol{\gamma}$ (gain) and $\boldsymbol{\beta}$ (bias) are learned per feature.

It keeps activations in a steady range as layers stack up. Whether it is placed before the sub-layer (Pre-LN) or after the residual addition (Post-LN) affects whether deep models need a learning-rate warmup to train.