ℝ^d — all vectors of length d
Read "R d" or "R to the d". It is the set of all lists of d real numbers, so "x ∈ ℝ^d" says x is a vector of d numbers.
The raised is the length of the list, not a power. is every possible vector of 768 numbers. Writing tells you the shape of (768 numbers) and nothing about what those numbers are.
When the raised part has two sizes joined by , it describes a matrix: is every table with rows and columns. The course states every tensor shape this way.
Example
Four slots, each holding any real number. In PyTorch this is a tensor of shape (4,).
Where the course uses it
- 1.1 Scalars, Vectors, Matrices, and Tensors · Module 1, Shapes and Linear Algebrafirst use
- 1.5 Embeddings: From a Token to a Vector · Module 1, Shapes and Linear Algebra
- 1.6 Linear Projections: One Vector Into Three · Module 1, Shapes and Linear Algebra
- 6.1 The Question Attention Answers · Module 6, Attention
- 6.6 Multi-Head Attention · Module 6, Attention
- 7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
- 7.5 The Output Layer · Module 7, The Block and the Full Model
- 8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 12.2 The KV Cache · Module 12, Generation
- 25.2 The Vision Transformer · Module 25, Vision Transformers and Contrastive Learning
- 26.1 Connecting Vision to Language · Module 26, Vision-Language Models
- 26.4 Training on Images and Text Together · Module 26, Vision-Language Models
- 28.3 Vector Quantization and Residual VQ · Module 28, Generating Audio: Codecs and TTS
Related terms
- ∈ — is inRead "is in" or "belongs to". The thing on the left is a member of the set on the right.
- ℝ — the real numbersRead "R" or "the reals". It is the set of all ordinary numbers: whole, negative, fractions and decimals.
- d_model — the model's widthRead "d model". It is how many numbers represent one token inside the network: 768 for GPT-2 small, 4096 for LLaMA-3 8B.