\sqrt{\;}

√ — square root

Read "the square root of". It is the non-negative number that, multiplied by itself, gives the number inside: √9 = 3.

In this course it mostly undoes a squaring. Variance is measured in squared units, so $\sqrt{\operatorname{Var}}$ brings the spread back to the scale of the data. That is why attention divides by $\sqrt{d_k}$ and not by $d_k$ : the score's variance is $d_k$ , so its typical size is $\sqrt{d_k}$ .

Example

\sqrt{64} = 8

Because 8 × 8 = 64.

Where the course uses it

3.1 Random Variables, Distributions, and Sampling · Module 3, Probability and Informationfirst use
3.2 Expectation and Variance · Module 3, Probability and Information
4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
6.1 The Question Attention Answers · Module 6, Attention
6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
6.3 Why Divide by the Square Root of the Key Width · Module 6, Attention
7.3 LayerNorm · Module 7, The Block and the Full Model
7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
10.2 AdamW · Module 10, Pretraining
10.4 Keeping Training Stable · Module 10, Pretraining
11.6 Activation Checkpointing · Module 11, Training Across Many GPUs
12.2 The KV Cache · Module 12, Generation
13.6 How Sure Is a Score? · Module 13, Evaluation
14.5 Superposition · Module 14, What the Model Learns
15.2 Stretching RoPE, and ALiBi · Module 15, The LLaMA Block
15.3 RMSNorm · Module 15, The LLaMA Block
16.4 Sandwich Normalization and Embedding Scaling · Module 16, Efficient and Long Attention
16.5 Logit Soft-Capping and QK-Norm · Module 16, Efficient and Long Attention
20.2 The Singular Value Decomposition · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
24.3 Outlier Channels and SmoothQuant · Module 24, Quantization and Number Formats

Related terms

Norm — the length of a vectorRead "the norm of x". Double bars mean the length of a vector: square each entry, add them up, and take the square root.
Var — varianceRead "the variance of X". It measures how spread out a random quantity is around its average. Its square root is the standard deviation.
The 1/√d_k scalingRead "one over root d k". Attention divides its scores by the square root of the head width so they keep a spread of about 1, however wide the head is.