# Module 24: Quantization and Number Formats

> How floats are stored (fp16, bf16, FP8), integer quantization, outlier channels and SmoothQuant, AWQ and GPTQ, and quantization-aware training.

- Part VI: Deployment · elective · Advanced · 4 hours
- By the end you can quantize a model and predict where the accuracy goes.
- Builds on: [Module 23: Serving](https://transformers-and-math.drmhse.com/md/23-serving.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/24-quantization
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

Decode is limited by reading weights and cache from memory (section 23.2), so fewer bits make generation faster as well as smaller. This module starts with how floats are laid out in bits, which explains section 4.3's overflow and the choice between fp16 and bf16, then covers integer quantization, the outlier problem of large models and its fixes, and the 8-bit and 4-bit float formats of recent hardware.

## 24.1 How Floating-Point Numbers Are Stored

A floating-point number is stored like scientific notation in base 2: a sign, an *exponent* that sets the scale, and a *mantissa* (or significand) that sets the digits. Its value is

$$
x = (-1)^{s} \times 1.m \times 2^{e - \text{bias}}
$$

where $s$ is the sign bit, $1.m$ the binary number "one point mantissa bits", and $e$ the stored exponent, offset by a fixed *bias* so negative powers of 2 can be stored. **Exponent bits set the range**; **mantissa bits set the precision**: with $m$ of them, neighbouring numbers near 1 are $2^{-m}$ apart.

| format | bits | sign | exponent | mantissa | largest value | spacing near 1 |
|---|---|---|---|---|---|---|
| fp32 | 32 | 1 | 8 | 23 | about $3.4 \times 10^{38}$ | $1.2 \times 10^{-7}$ |
| fp16 | 16 | 1 | 5 | 10 | 65,504 | $9.8 \times 10^{-4}$ |
| bf16 | 16 | 1 | 8 | 7 | about $3.4 \times 10^{38}$ | $7.8 \times 10^{-3}$ |

fp16's 5 exponent bits end its range at 65,504, so $e^{z}$ overflows past $z \approx 11$ (section 4.3). bf16 ("brain float") keeps fp32's 8 exponent bits and gives up mantissa instead. For training, range matters more than precision: an overflow is a `NaN` that destroys the run, while imprecision is noise gradient descent tolerates. So bf16 became the standard training format, with an fp32 master copy for small updates its coarse spacing would round away (section 10.4).

> **Interactive figure** `float-bits`, on the web page: https://transformers-and-math.drmhse.com/modules/24-quantization#figure-float-bits
>
> **Try:** Take the number 1.0 in bf16 and add the smallest possible step. Before toggling the last mantissa bit, predict the next representable number after 1.0, and why adding 0.001 to 1.0 in bf16 does nothing.
>
> **What happens:** The next number is $1 + 2^{-7} = 1.0078125$. Anything smaller than half that step rounds back to 1.0, so $1.0 + 0.001$ is still exactly $1.0$ in bf16. A tiny weight update applied directly to bf16 weights is lost, which is why training keeps an fp32 copy.

> **In words.** One number's bits, coloured as sign, exponent and mantissa, in fp32, fp16, bf16 or the two 8-bit float formats. Clicking a bit flips it; the readout decodes the value and gives the distance to the next representable number.

## 24.2 Uniform Integer Quantization

Weights stored in 16 bits can instead be stored as 8-bit integers (INT8), 4-bit integers (INT4) or 8-bit floats (FP8), trading precision for memory and speed. Integer quantization maps reals in $[x_{\min}, x_{\max}]$ onto a $b$-bit integer range $[q_{\min}, q_{\max}]$, such as $[-8, 7]$ for INT4 or $[-128, 127]$ for INT8:

1. **Choose a scale $s$**, the real value of one integer step, **and a zero-point $z$**, the integer standing for $0$.
   $$
   s = \frac{x_{\max} - x_{\min}}{2^b - 1}, \quad z = \operatorname{round}\left(-\frac{x_{\min}}{s}\right) + q_{\min}
   $$
2. **Quantize**: divide, round to nearest ($\lfloor \cdot \rceil$), add the zero-point, and clip to the range.
   $$
   q = \operatorname{clamp}\left(\left\lfloor \frac{x}{s} \right\rceil + z, \, q_{\min}, \, q_{\max}\right)
   $$
3. **Dequantize** to approximate the original:
   $$
   \hat{x} = s \cdot (q - z) \approx x
   $$

> **Interactive figure** `quant-number-line`, on the web page: https://transformers-and-math.drmhse.com/modules/24-quantization#figure-quant-number-line
>
> **Try:** With 4 bits there are 16 levels. Before raising the outlier slider, predict what happens to the rounding error of the ordinary values when one value in the tensor becomes ten times larger than the rest.
>
> **What happens:** The error of every ordinary value grows. One scale covers the whole range, so the outlier stretches the range, the levels spread out, and the ordinary values, crowded near zero, share a few levels. This is the outlier problem of section 24.3, on a single row of numbers.

> **In words.** A row of values on a number line, each joined to the grid level it snaps to. Sliders set the bit width and one outlier's size; the readout gives the scale, the number of levels and the ordinary values' average rounding error.

For example, with INT8 and $x \in [-1, 3]$: $s = 4/255 \approx 0.0157$ and $z = \operatorname{round}(63.75) - 128 = -64$. The value $x = 0.5$ becomes $q = 32 - 64 = -32$, and dequantizes to $32$ steps of $0.0157$, about $0.502$. The error is $0.002$, and for any value inside the range it is at most $s/2$.

- **Symmetric quantization** uses $[-\max|x|, \max|x|]$, with $s = \max|x| / q_{\max}$ and $z = 0$, so integer matrix multiplies need no correction terms. It is usual for weights, spread roughly evenly around zero.
- **Asymmetric quantization** uses the true minimum and maximum, suiting lopsided values: ReLU outputs are never negative and GELU's never below about $-0.17$, so a symmetric range would waste almost half its integers.

## 24.3 Outlier Channels and SmoothQuant

In LLMs from about 6.7B parameters up (Dettmers et al., 2022), roughly $0.1\%$ of activation **channels** (columns of the activation matrix $X$) carry values up to $100\times$ larger than typical, the same channels on every token. One scale for the whole tensor must then be large, crowding the other $99.9\%$ into a few integer levels.

**SmoothQuant** (Xiao et al., 2023) moves the difficulty into the weights. $Y = X W$ is unchanged if each input channel of $X$ is divided by a factor and the matching row of $W$ multiplied by it:

$$
Y = X W = \left(X \operatorname{diag}(\mathbf{s})^{-1}\right) \left(\operatorname{diag}(\mathbf{s}) W\right) = \hat{X} \hat{W}
$$

The per-channel factors $\mathbf{s}$ (unrelated to the quantization scale $s$) form a diagonal matrix (section 1.3). SmoothQuant picks $s_j = \max(|X_j|)^\alpha / \max(|W_j|)^{1-\alpha}$, with column $X_j$ measured on sample inputs, row $W_j$ of the weights, and usually $\alpha = 0.5$. If channel $j$'s activations reach $50$ and its weights $0.5$, then $s_j = \sqrt{50} / \sqrt{0.5} = 10$, and both sides reach $5$: the outlier is split evenly.

$\operatorname{diag}(\mathbf{s}) W$ is computed once before deployment, and the division of $X$ is folded into the preceding normalization layer, so inference does no extra work. Both weights and activations then quantize to INT8 (W8A8) with little accuracy loss.

> **Interactive figure** `quant-outliers`, on the web page: https://transformers-and-math.drmhse.com/modules/24-quantization#figure-quant-outliers
>
> **Try:** Before applying SmoothQuant, predict where the two outlier activation channels' size goes, since the product $XW$ must not change.
>
> **What happens:** Into the matching rows of the weights. Dividing a channel of $X$ by $s_j$ and multiplying row $j$ of $W$ by $s_j$ leaves $XW$ exactly the same, but now neither side has an extreme channel, so both quantize well.

> **In words.** Activations as a tokens-by-channels heatmap beside the weights. Two channels are about forty times larger on every token, leaving ordinary values few 8-bit levels. After SmoothQuant, the outliers shrink, the weights take on some of the size, and ordinary values spread over many more levels.

## 24.4 Post-Training Quantization: AWQ and GPTQ

*Post-training quantization* (PTQ) quantizes a trained model without further training. Rounding each weight independently works well at 8 bits, but at 4 bits the errors add up. Two methods do much better, using a small *calibration set* of sample inputs to see which errors matter.

- **AWQ (Activation-aware Weight Quantization)**: about $1\%$ of weight channels matter far more than the rest, found by their large input activations. Keeping them in 16 bits would be slow on real hardware, so AWQ scales them up before quantizing, and the matching activations down (SmoothQuant's trick), giving them relatively smaller rounding error while every weight is stored in INT4.
- **GPTQ** rounds a layer's weights one input dimension at a time, adjusting the still-unrounded weights to cancel each rounding error. It uses the **Hessian**, the matrix of second derivatives $\partial^2 E / \partial w_i \partial w_j$ of the output error: how *curved* the error is in each direction and how pairs of weights interact. For a linear layer with squared error it is $H = 2X^\top X$ over the calibration inputs: a large diagonal entry means a sensitive weight, a large off-diagonal one means two weights' errors can partly cancel. After rounding weight $i$ of the vector $\mathbf{w}$ feeding one output from $w_i$ to $\hat{w}_i$:

$$
\mathbf{w}^* = \mathbf{w} - \frac{w_i - \hat{w}_i}{[H^{-1}]_{ii}} H^{-1}_{:, i}
$$

$H^{-1}_{:,i}$ is column $i$ of the inverse Hessian and $[H^{-1}]_{ii}$ its diagonal entry. Dividing by it sets weight $i$ exactly to $\hat{w}_i$, while shifting every other weight according to how strongly it interacts with weight $i$.

## 24.5 Quantization-Aware Training

*Quantization-aware training* (QAT) simulates quantization during training or a short fine-tune, so the model learns weights that survive rounding. The forward pass quantizes and dequantizes each weight; the backward pass treats rounding as the identity, since its zero slope would otherwise block every gradient: the *straight-through estimator* (Bengio et al., 2013; Jacob et al., 2018). At 4 bits and below QAT recovers more accuracy than post-training methods, at the cost of training compute, and some model families release QAT checkpoints for 4-bit use.

## 24.6 FP8 and Microscaling

Recent NVIDIA GPUs (Hopper and Blackwell) compute directly in 8-bit floating point, in two layouts (section 24.1): **E4M3** (4 exponent, 3 mantissa bits) for precision, used for weights and activations, and **E5M2** (5 and 2) for range, used for gradients, whose sizes vary more. FP8 halves memory against 16 bits and roughly doubles peak matrix-multiply speed. Its range is still narrow, so each tensor carries its own scale factor. **Microscaling** formats (MXFP8 and MXFP4, on Blackwell) give each block of 32 values its own shared scale, so an outlier affects only its block.

> **Browser lab** `quantize-error`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/24-quantization#lab-quantize-error

## Exercises

**Exercise (check).** Why did bf16, rather than fp16, become the standard format for training?

*Solution:* bf16 keeps fp32's 8 exponent bits and so its range, giving up mantissa bits instead. For training, overflow (which fp16's 5-bit exponent makes likely) is fatal while a little imprecision is just noise.

**Exercise (apply).** Quantize $x = 0.5$ with INT8 on the range $[-1, 3]$ (asymmetric). What integer is stored?

*Answer:* -32 (within 0)

*Solution:* $s = 4/255 \approx 0.01569$ and $z = \operatorname{round}(1/s) - 128 = \operatorname{round}(63.75) - 128 = -64$. Then $q = \operatorname{round}(0.5/0.01569) + z = 32 - 64 = -32$.

**Exercise (apply).** What is the largest finite fp16 value?

*Answer:* 65504 (within 0)

*Solution:* The largest exponent below the reserved all-ones pattern gives $2^{15}$, and the largest mantissa $1.1111111111_2 = 2 - 2^{-10}$. So $2^{15} \times (2 - 2^{-10}) = 65{,}504$.

**Exercise (apply).** SmoothQuant with $\alpha = 0.5$: channel $j$'s activations reach 64 and its weights reach 0.25. What is $s_j$?

*Answer:* 16 (within 0)

*Solution:* $s_j = \max|X_j|^{0.5} / \max|W_j|^{0.5} = 8 / 0.5 = 16$. After scaling, the activations reach $64/16 = 4$ and the weights $0.25 \times 16 = 4$: the outlier is shared evenly.

**Exercise (apply).** How many bytes does a 7-billion-parameter model take at 4 bits per weight, ignoring the scales, in GB?

*Answer:* 3.5 (within 0.01)

*Solution:* $7 \times 10^9 \times 0.5$ bytes $= 3.5$ GB, against 14 GB in bf16. The per-group scales add a few percent.

**Exercise (extend).** Why is rounding a weight a problem for training, and how does the straight-through estimator get around it?

*Solution:* Rounding is a staircase: its derivative is zero almost everywhere (and undefined at the steps), so backpropagating through it gives zero gradient and nothing learns. The straight-through estimator uses the rounded value in the forward pass but treats rounding as the identity in the backward pass, passing the gradient through unchanged, so the underlying full-precision weights still receive useful updates.

### Core Academic References
- **IEEE (2019)**: *IEEE Standard for Floating-Point Arithmetic* (IEEE 754-2019).
- **Kalamkar et al. (2019)**: *A Study of BFLOAT16 for Deep Learning Training*. [arXiv:1905.12322](https://arxiv.org/abs/1905.12322)
- **Jacob et al. (2018)**: *Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference*. CVPR 2018. [arXiv:1712.05877](https://arxiv.org/abs/1712.05877)
- **Dettmers et al. (2022)**: *LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale*. NeurIPS 2022. [arXiv:2208.07339](https://arxiv.org/abs/2208.07339)
- **Xiao et al. (2023)**: *SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models*. ICML 2023. [arXiv:2211.10438](https://arxiv.org/abs/2211.10438)
- **Lin et al. (2023)**: *AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration*. [arXiv:2306.00978](https://arxiv.org/abs/2306.00978)
- **Frantar et al. (2022)**: *GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers*. [arXiv:2210.17323](https://arxiv.org/abs/2210.17323)
- **Bengio, Léonard & Courville (2013)**: *Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation* (straight-through estimator). [arXiv:1308.3432](https://arxiv.org/abs/1308.3432)
- **Micikevicius et al. (2022)**: *FP8 Formats for Deep Learning*. [arXiv:2209.05433](https://arxiv.org/abs/2209.05433)
- **Rouhani et al. (2023)**: *Microscaling Data Formats for Deep Learning*. [arXiv:2310.10537](https://arxiv.org/abs/2310.10537)