\text{cat} \to 3797

Token

One piece of text the model reads or writes: a word, part of a word, or a punctuation mark. Each token has a whole-number ID in the vocabulary.

A tokenizer splits text into tokens and looks each one up in the vocabulary, giving a whole number. The model then replaces each number with a vector from its embedding matrix (section 1.5). Common words are usually one token; rare words are split into several (section 9.2).

A sentence of $N$ tokens therefore becomes $N$ vectors, stacked as the rows of an $N \times d_{\text{model}}$ matrix.

Example

\text{unbelievable} \to [\text{un},\; \text{believ},\; \text{able}]

One word, three tokens (illustrative split; each tokenizer splits differently).

Where the course uses it

1.5 Embeddings: From a Token to a Vector · Module 1, Shapes and Linear Algebrafirst use

Related terms

|V| — size of the vocabularyRead "the size of V". Single bars around a set mean "how many items it has", so |V| is the number of tokens in the vocabulary.
eᵢ — a one-hot vectorRead "e i". It is a vector with a 1 in position i and 0 everywhere else: token number i, written as a vector.
W_E — the embedding matrixRead "W E". It is a learned table with one row per token in the vocabulary; each row is that token's vector of d_model numbers.
N, T, L — sequence lengthThe number of tokens in the sequence. Different modules call it N, T or L. N² ("N squared") is how attention's cost grows with it.