Data type — how each number is stored
The format each number in a tensor is stored in, which sets its size in bytes and its precision. fp32 uses 4 bytes, bf16 and fp16 use 2, int8 uses 1.
A computer cannot store an arbitrary real number, so each entry is rounded to a fixed format. Floating-point formats keep a sign, an exponent (the range) and a mantissa (the precision): fp32 is standard, while bf16 and fp16 halve the memory. Integer formats such as int8 store whole numbers plus a shared scale, the subject of section 24.2.
The memory of a tensor is its number of entries times the bytes per entry. That is why the same model can need 4, 2 or 1 byte per weight depending on its data type.
Example
GPT-2's token embedding matrix in fp32. In bf16 it is about 77 MB.
Where the course uses it
Related terms
- Integer quantizationConverts a real number x into a small integer q: divide by a scale s, round to the nearest integer, add a zero-point z, and clip to the allowed range.
- Weights (parameters)The numbers a model learns in training: the entries of its weight matrices and vectors. A model's size is counted in weights, such as 124 million for GPT-2 Small.
- Tensor — numbers on any number of axesAn array of numbers with any number of axes. In code every array is called a tensor: a scalar has 0 axes, a vector 1, a matrix 2. In prose the word usually means 3 or more.