≈, ∼, ∝ — approximately, distributed as, proportional to
≈ reads "approximately equals". ∼ reads "is distributed as" or "is roughly of the order of". ∝ reads "is proportional to".
usually marks a deliberate approximation, such as the tanh formula for GELU or a rounded parameter count. after quantization admits that the scheme introduces some error.
reads " is drawn from a normal distribution with mean 0 and variance 1". means for some fixed constant .
Example
The second says softmax probabilities equal e to the z times a shared constant (one over the sum).
Where the course uses it
- 2.1 Derivatives: Slopes and the Rules for Finding Them · Module 2, Calculus and Optimizationfirst use
- 2.3 The Chain Rule and the Jacobian · Module 2, Calculus and Optimization
- 2.5 Backpropagation · Module 2, Calculus and Optimization
- 2.6 What Goes Wrong in Long Chains · Module 2, Calculus and Optimization
- 3.1 Random Variables, Distributions, and Sampling · Module 3, Probability and Information
- 4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
- 5.4 Vanishing and Exploding Gradients · Module 5, Sequences Before Transformers
- 6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
- 6.3 Why Divide by the Square Root of the Key Width · Module 6, Attention
- 7.3 LayerNorm · Module 7, The Block and the Full Model
- 7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
- 7.6 The Complete Model · Module 7, The Block and the Full Model
- 10.5 Scaling Laws: How Big, and on How Much Text · Module 10, Pretraining
- 11.1 Why One GPU Is Not Enough · Module 11, Training Across Many GPUs
- 13.6 How Sure Is a Score? · Module 13, Evaluation
- 15.4 SwiGLU · Module 15, The LLaMA Block
- 15.6 The 2017 Block and the 2024 Block, Side by Side · Module 15, The LLaMA Block
- 16.3 Local and Global Attention · Module 16, Efficient and Long Attention
- 16.5 Logit Soft-Capping and QK-Norm · Module 16, Efficient and Long Attention
- 17.2 Many Small Experts · Module 17, Mixture of Experts
- 21.1 A Reinforcement Learning Primer · Module 21, Preference Learning: RL, RLHF and DPO
- 21.2 Policy Gradients · Module 21, Preference Learning: RL, RLHF and DPO
- 21.5 RLHF End to End · Module 21, Preference Learning: RL, RLHF and DPO
- 24.1 How Floating-Point Numbers Are Stored · Module 24, Quantization and Number Formats
- 24.2 Uniform Integer Quantization · Module 24, Quantization and Number Formats
Related terms
- Hat and tilde — an estimate or modified versionRead x̂ as "x hat" and x̃ as "x tilde". Both mark a changed copy of x: most often an estimate or a rounded version (hat), or a transformed version (tilde).
- GELU, ReLU, Swish — activation functionsThe nonlinearity applied inside the feed-forward block. ReLU sets negative numbers to zero; GELU and Swish do almost the same but with a smooth curve.