|V|

|V| — size of the vocabulary

Read "the size of V". Single bars around a set mean "how many items it has", so |V| is the number of tokens in the vocabulary.

$V$ is the vocabulary: the fixed list of tokens the tokenizer can produce. $|V|$ counts them. It is 50,257 for GPT-2, 32,000 for LLaMA-2 and 256,000 for Gemma.

Do not confuse it with two similar notations. $\|\mathbf{v}\|$ (double bars) is the length of a vector. $|x|$ on a single number is its absolute value, so $|-3| = 3$ .

Example

V = \{\text{the}, \text{cat}, \text{sat}\} \;\Longrightarrow\; |V| = 3

A toy vocabulary of three tokens.

Where the course uses it

0.1 From a Prompt to a Next-Token Probability · Module 0, Start Here: What an LLM Doesfirst use
1.5 Embeddings: From a Token to a Vector · Module 1, Shapes and Linear Algebra
3.1 Random Variables, Distributions, and Sampling · Module 3, Probability and Information
7.5 The Output Layer · Module 7, The Block and the Full Model
7.6 The Complete Model · Module 7, The Block and the Full Model
9.4 Choosing a Vocabulary Size · Module 9, Tokenization

Related terms

Norm — the length of a vectorRead "the norm of x". Double bars mean the length of a vector: square each entry, add them up, and take the square root.
eᵢ — a one-hot vectorRead "e i". It is a vector with a 1 in position i and 0 everywhere else: token number i, written as a vector.