θ — the model's parameters (or an angle)
Read "theta". Usually it stands for every learnable weight in the model at once. In some sections it is an angle or a setting instead; the sentence around it tells you which.
means "the probability that this model, with its current weights , gives to ". The subscript is a reminder that the probabilities change as training changes the weights.
has several meanings in this course. In Module 1 it is the angle between two vectors. In RoPE (Module 15) it is a rotation frequency. In the repetition penalty (Module 8) it is a single number you choose, such as 1.15. Read the surrounding sentence to know which.
Where the course uses it
- 1.4 Dot Products, Length, and Angle · Module 1, Shapes and Linear Algebrafirst use
- 2.2 Partial Derivatives and the Gradient · Module 2, Calculus and Optimization
- 2.4 Gradient Descent · Module 2, Calculus and Optimization
- 3.4 Maximum Likelihood · Module 3, Probability and Information
- 8.2 Encoder-Only: BERT and Masked Language Modeling · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 10.2 AdamW · Module 10, Pretraining
- 12.3 Decoding: From Logits to Text · Module 12, Generation
- 13.1 Perplexity and Its Limits · Module 13, Evaluation
- 15.1 Rotary Position Embedding (RoPE) · Module 15, The LLaMA Block
- 15.2 Stretching RoPE, and ALiBi · Module 15, The LLaMA Block
- 19.1 Comparing the Architectures · Module 19, Reading a Model Card
- 20.1 Supervised Instruction Tuning · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
- 21.1 A Reinforcement Learning Primer · Module 21, Preference Learning: RL, RLHF and DPO
- 21.2 Policy Gradients · Module 21, Preference Learning: RL, RLHF and DPO
- 21.3 PPO · Module 21, Preference Learning: RL, RLHF and DPO
- 21.5 RLHF End to End · Module 21, Preference Learning: RL, RLHF and DPO
- 21.6 Direct Preference Optimization · Module 21, Preference Learning: RL, RLHF and DPO
- 26.4 Training on Images and Text Together · Module 26, Vision-Language Models
Related terms
- 𝓛 — the lossRead "L" or "the loss". The curly L is the single number that measures how wrong the model is. Training adjusts the weights to make it small.
- RoPE rotation matrixRead "R theta m". It rotates a query or key vector by an angle that grows with the token's position m, so position is stored as an amount of rotation.