W_E — the embedding matrix

$W_E \in \mathbb{R}^{|V| \times d_{\text{model}}}$: $|V|$ rows, each $d_{\text{model}}$ numbers wide. Row $i$ is the vector the model uses for token $i$. It is trained like any other weight.

Many models tie this matrix to the output layer: they reuse it (transposed) to turn the final hidden vector back into one score per vocabulary token. That saves a whole $|V| \times d_{\text{model}}$ block of parameters, which is a large share of a small model.