Weights (parameters)
The numbers a model learns in training: the entries of its weight matrices and vectors. A model's size is counted in weights, such as 124 million for GPT-2 Small.
Everything else a model computes is derived from its input and its weights. The weights are fixed while the model runs and change only during training. The words weights and parameters mean the same thing; stands for all of them at once.
Memory is weights times bytes per number: 124 million weights stored as 16-bit numbers take about 248 MB.
Example
One small matrix of one attention head in GPT-2 Small.
Where the course uses it
Related terms
- θ — the model's parameters (or an angle)Read "theta". Usually it stands for every learnable weight in the model at once. In some sections it is an angle or a setting instead; the sentence around it tells you which.
- TrainingThe process that sets a model's weights. It shows the model examples, measures how wrong it is with the loss, and nudges every weight to make the loss smaller. Repeated millions of times.
- Matrix — a table of numbersA rectangular table of numbers, with two axes: rows and columns. An m×n matrix has m rows and n columns. Written as a capital letter, such as W.
- Data type — how each number is storedThe format each number in a tensor is stored in, which sets its size in bytes and its precision. fp32 uses 4 bytes, bf16 and fp16 use 2, int8 uses 1.