Quantization and Number Formats

Decode is limited by reading weights and cache from memory (section 23.2), so fewer bits make generation faster as well as smaller. This module starts with how floats are laid out in bits, which explains section 4.3's overflow and the choice between fp16 and bf16, then covers integer quantization, the outlier problem of large models and its fixes, and the 8-bit and 4-bit float formats of recent hardware.

24.1 How Floating-Point Numbers Are StoredRun this section

A floating-point number is stored like scientific notation in base 2: a sign, an exponent that sets the scale, and a mantissa (or significand) that sets the digits. Its value is

where $s$ is the sign bit, $1.m$ the binary number "one point mantissa bits", and $e$ the stored exponent, offset by a fixed bias so negative powers of 2 can be stored. Exponent bits set the range; mantissa bits set the precision: with $m$ of them, neighbouring numbers near 1 are $2^{-m}$ apart.

fp16's 5 exponent bits end its range at 65,504, so $e^{z}$ overflows past $z \approx 11$ (section 4.3). bf16 ("brain float") keeps fp32's 8 exponent bits and gives up mantissa instead. For training, range matters more than precision: an overflow is a NaN that destroys the run, while imprecision is noise gradient descent tolerates. So bf16 became the standard training format, with an fp32 master copy for small updates its coarse spacing would round away (section 10.4).

In words. One number's bits, coloured as sign, exponent and mantissa, in fp32, fp16, bf16 or the two 8-bit float formats. Clicking a bit flips it; the readout decodes the value and gives the distance to the next representable number.

24.2 Uniform Integer QuantizationRun this section

Weights stored in 16 bits can instead be stored as 8-bit integers (INT8), 4-bit integers (INT4) or 8-bit floats (FP8), trading precision for memory and speed. Integer quantization maps reals in $[x_{\min}, x_{\max}]$ onto a $b$-bit integer range $[q_{\min}, q_{\max}]$, such as $[-8, 7]$ for INT4 or $[-128, 127]$ for INT8:

1. Choose a scale $s$, the real value of one integer step, and a zero-point $z$, the integer standing for $0$.
   $$s = \frac{x_{\max} - x_{\min}}{2^b - 1}, \quad z = \operatorname{round}\left(-\frac{x_{\min}}{s}\right) + q_{\min}$$

   What does this say?
2. Quantize: divide, round to nearest ($\lfloor \cdot \rceil$), add the zero-point, and clip to the range.
   $$q = \operatorname{clamp}\left(\left\lfloor \frac{x}{s} \right\rceil + z, \, q_{\min}, \, q_{\max}\right)$$

   What does this say?
3. Dequantize to approximate the original:
   $$\hat{x} = s \cdot (q - z) \approx x$$

   What does this say?introduces 1 notation

In words. A row of values on a number line, each joined to the grid level it snaps to. Sliders set the bit width and one outlier's size; the readout gives the scale, the number of levels and the ordinary values' average rounding error.

For example, with INT8 and $x \in [-1, 3]$: $s = 4/255 \approx 0.0157$ and $z = \operatorname{round}(63.75) - 128 = -64$. The value $x = 0.5$ becomes $q = 32 - 64 = -32$, and dequantizes to $32$ steps of $0.0157$, about $0.502$. The error is $0.002$, and for any value inside the range it is at most $s/2$.

• Symmetric quantization uses $[-\max|x|, \max|x|]$, with $s = \max|x| / q_{\max}$ and $z = 0$, so integer matrix multiplies need no correction terms. It is usual for weights, spread roughly evenly around zero.
• Asymmetric quantization uses the true minimum and maximum, suiting lopsided values: ReLU outputs are never negative and GELU's never below about $-0.17$, so a symmetric range would waste almost half its integers.

24.3 Outlier Channels and SmoothQuantRun this section

In LLMs from about 6.7B parameters up (Dettmers et al., 2022), roughly $0.1\%$ of activation channels (columns of the activation matrix $X$) carry values up to $100\times$ larger than typical, the same channels on every token. One scale for the whole tensor must then be large, crowding the other $99.9\%$ into a few integer levels.

SmoothQuant (Xiao et al., 2023) moves the difficulty into the weights. $Y = X W$ is unchanged if each input channel of $X$ is divided by a factor and the matching row of $W$ multiplied by it:

The per-channel factors $\mathbf{s}$ (unrelated to the quantization scale $s$) form a diagonal matrix (section 1.3). SmoothQuant picks $s_j = \max(|X_j|)^\alpha / \max(|W_j|)^{1-\alpha}$, with column $X_j$ measured on sample inputs, row $W_j$ of the weights, and usually $\alpha = 0.5$. If channel $j$'s activations reach $50$ and its weights $0.5$, then $s_j = \sqrt{50} / \sqrt{0.5} = 10$, and both sides reach $5$: the outlier is split evenly.

$\operatorname{diag}(\mathbf{s}) W$ is computed once before deployment, and the division of $X$ is folded into the preceding normalization layer, so inference does no extra work. Both weights and activations then quantize to INT8 (W8A8) with little accuracy loss.

In words. Activations as a tokens-by-channels heatmap beside the weights. Two channels are about forty times larger on every token, leaving ordinary values few 8-bit levels. After SmoothQuant, the outliers shrink, the weights take on some of the size, and ordinary values spread over many more levels.

24.4 Post-Training Quantization: AWQ and GPTQ

Post-training quantization (PTQ) quantizes a trained model without further training. Rounding each weight independently works well at 8 bits, but at 4 bits the errors add up. Two methods do much better, using a small calibration set of sample inputs to see which errors matter.

• AWQ (Activation-aware Weight Quantization): about $1\%$ of weight channels matter far more than the rest, found by their large input activations. Keeping them in 16 bits would be slow on real hardware, so AWQ scales them up before quantizing, and the matching activations down (SmoothQuant's trick), giving them relatively smaller rounding error while every weight is stored in INT4.
• GPTQ rounds a layer's weights one input dimension at a time, adjusting the still-unrounded weights to cancel each rounding error. It uses the Hessian, the matrix of second derivatives $\partial^2 E / \partial w_i \partial w_j$ of the output error: how curved the error is in each direction and how pairs of weights interact. For a linear layer with squared error it is $H = 2X^\top X$ over the calibration inputs: a large diagonal entry means a sensitive weight, a large off-diagonal one means two weights' errors can partly cancel. After rounding weight $i$ of the vector $\mathbf{w}$ feeding one output from $w_i$ to $\hat{w}_i$:

$H^{-1}_{:,i}$ is column $i$ of the inverse Hessian and $[H^{-1}]_{ii}$ its diagonal entry. Dividing by it sets weight $i$ exactly to $\hat{w}_i$, while shifting every other weight according to how strongly it interacts with weight $i$.

24.5 Quantization-Aware TrainingRun this section

Quantization-aware training (QAT) simulates quantization during training or a short fine-tune, so the model learns weights that survive rounding. The forward pass quantizes and dequantizes each weight; the backward pass treats rounding as the identity, since its zero slope would otherwise block every gradient: the straight-through estimator (Bengio et al., 2013; Jacob et al., 2018). At 4 bits and below QAT recovers more accuracy than post-training methods, at the cost of training compute, and some model families release QAT checkpoints for 4-bit use.

24.6 FP8 and Microscaling

Recent NVIDIA GPUs (Hopper and Blackwell) compute directly in 8-bit floating point, in two layouts (section 24.1): E4M3 (4 exponent, 3 mantissa bits) for precision, used for weights and activations, and E5M2 (5 and 2) for range, used for gradients, whose sizes vary more. FP8 halves memory against 16 bits and roughly doubles peak matrix-multiply speed. Its range is still narrow, so each tensor carries its own scale factor. Microscaling formats (MXFP8 and MXFP4, on Blackwell) give each block of 32 values its own shared scale, so an outlier affects only its block.

Exercises

Core Academic References

• IEEE (2019): IEEE Standard for Floating-Point Arithmetic (IEEE 754-2019).
• Kalamkar et al. (2019): A Study of BFLOAT16 for Deep Learning Training. arXiv:1905.12322
• Jacob et al. (2018): Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR 2018. arXiv:1712.05877
• Dettmers et al. (2022): LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022. arXiv:2208.07339
• Xiao et al. (2023): SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023. arXiv:2211.10438
• Lin et al. (2023): AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978
• Frantar et al. (2022): GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323
• Bengio, Léonard & Courville (2013): Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (straight-through estimator). arXiv:1308.3432
• Micikevicius et al. (2022): FP8 Formats for Deep Learning. arXiv:2209.05433
• Rouhani et al. (2023): Microscaling Data Formats for Deep Learning. arXiv:2310.10537