∂ — partial derivative
Read "partial L by partial θ". It is how much the top quantity changes when you nudge the bottom one slightly, keeping every other input fixed.
is the slope of the loss with respect to one parameter. Its sign tells training which way to move that parameter, and its size tells how much the loss cares. The curly (instead of the in ) signals that the function has several inputs and all the others are held still.
When the top and bottom are both vectors, the result is a matrix holding the slope of every output with respect to every input. That matrix is called a Jacobian.
Example
The softmax Jacobian. Every output p_i depends on every input z_j, because all outputs share the same denominator.
Where the course uses it
- 2.2 Partial Derivatives and the Gradient · Module 2, Calculus and Optimizationfirst use
- 2.3 The Chain Rule and the Jacobian · Module 2, Calculus and Optimization
- 2.5 Backpropagation · Module 2, Calculus and Optimization
- 4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
- 5.4 Vanishing and Exploding Gradients · Module 5, Sequences Before Transformers
- 7.2 Residual Connections and the Residual Stream · Module 7, The Block and the Full Model
- 8.5 Where the Norm Goes: Post-LN and Pre-LN · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 24.4 Post-Training Quantization: AWQ and GPTQ · Module 24, Quantization and Number Formats
Related terms
- ∇ — gradientRead "nabla" or "the gradient of". It is the list of every partial derivative at once, and it points in the direction that increases the function fastest.
- δᵢⱼ — Kronecker deltaRead "delta i j". It equals 1 when i and j are the same number and 0 when they differ. It is an if-statement written as a number.
- Gradient through many stepsHow much the state at the last step T depends on the state at an earlier step t. The chain rule makes it a product of one factor per step, which is why RNN gradients vanish or explode.