\lambda

λ — weight decay, or a context-extension factor

Read "lambda". In AdamW (section 10.2) it is the weight-decay strength that pulls weights toward zero. In section 15.1 it is how many times longer a stretched context is.

The two uses are unrelated. In the AdamW update, $\lambda \theta$ shrinks every weight a little at each step; a typical value is $0.1$ . In NTK-aware scaling, $\lambda$ is the new context length divided by the trained one, so going from 8k to 64k tokens means $\lambda = 8$ .

Where the course uses it

10.2 AdamW · Module 10, Pretrainingfirst use
15.2 Stretching RoPE, and ALiBi · Module 15, The LLaMA Block

Related terms

η — the learning rateRead "eta". The learning rate: how big a step gradient descent takes against the gradient on each update.