∈ — is in
Read "is in" or "belongs to". The thing on the left is a member of the set on the right.
says that is one of the things in the set . In this course it mostly does one job: stating a shape. means " is a vector of 64 real numbers", like a type annotation in code.
It is also used with a written-out set: says is one of the whole numbers in that range.
Example
i is a token id: a whole number from 0 up to one less than the vocabulary size.
Where the course uses it
- 1.1 Scalars, Vectors, Matrices, and Tensors · Module 1, Shapes and Linear Algebrafirst use
- 1.5 Embeddings: From a Token to a Vector · Module 1, Shapes and Linear Algebra
- 1.6 Linear Projections: One Vector Into Three · Module 1, Shapes and Linear Algebra
- 6.1 The Question Attention Answers · Module 6, Attention
- 6.6 Multi-Head Attention · Module 6, Attention
- 7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
- 7.5 The Output Layer · Module 7, The Block and the Full Model
- 8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 8.2 Encoder-Only: BERT and Masked Language Modeling · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 12.2 The KV Cache · Module 12, Generation
- 12.3 Decoding: From Logits to Text · Module 12, Generation
- 15.1 Rotary Position Embedding (RoPE) · Module 15, The LLaMA Block
- 17.2 Many Small Experts · Module 17, Mixture of Experts
- 24.2 Uniform Integer Quantization · Module 24, Quantization and Number Formats
- 25.2 The Vision Transformer · Module 25, Vision Transformers and Contrastive Learning
- 26.1 Connecting Vision to Language · Module 26, Vision-Language Models
- 26.4 Training on Images and Text Together · Module 26, Vision-Language Models
- 27.3 The CTC Objective and Its Recursion · Module 27, Recognition: OCR and Speech With CTC
- 28.3 Vector Quantization and Residual VQ · Module 28, Generating Audio: Codecs and TTS
Related terms
- ℝ^d — all vectors of length dRead "R d" or "R to the d". It is the set of all lists of d real numbers, so "x ∈ ℝ^d" says x is a vector of d numbers.
- { } — a setCurly braces list the members of a set: a collection of values with no order and no repeats. Read {0, 1} as "the set of 0 and 1".
- |V| — size of the vocabularyRead "the size of V". Single bars around a set mean "how many items it has", so |V| is the number of tokens in the vocabulary.