Integer quantization
Converts a real number x into a small integer q: divide by a scale s, round to the nearest integer, add a zero-point z, and clip to the allowed range.
The scale is how much real value one integer step stands for. The zero-point is the integer that stands for exactly 0.0. It must be exact, or every zero in the tensor (including padding) would come back slightly off.
The mixed brackets mean round to the nearest integer. clips the result into the range the integer type can hold, from to (for example to for 8-bit). To convert back, . In general , and keeping that error small is the whole design problem.
Example
The round trip loses 0.03.
Where the course uses it
Related terms
- Hat and tilde — an estimate or modified versionRead x̂ as "x hat" and x̃ as "x tilde". Both mark a changed copy of x: most often an estimate or a rounded version (hat), or a transformed version (tilde).
- Ceiling, floor, round⌈x⌉ reads "ceiling of x" and rounds up. ⌊x⌋ reads "floor of x" and rounds down. The mixed ⌊x⌉ rounds to the nearest whole number.