\sum

∑ — sum

Read "the sum of". The big sigma is a loop that adds: the counter starts at the value below it and stops at the value above it.

$\sum_{i=1}^{d} x_i$ means: set $i$ to 1 and take $x_1$ , set $i$ to 2 and take $x_2$ , and so on up to $x_d$ , then add them all.

In Python it is sum(x[i] for i in range(1, d + 1)). When there is a set or condition below instead of a start value, such as $\sum_{i \in S}$ , it means "add one term for every member of $S$ ". A bare $\sum_i$ means "over every value of $i$ ".

Example

\sum_{i=1}^{4} i^2 = 1 + 4 + 9 + 16 = 30

The counter i takes the values 1, 2, 3, 4; each is squared, then the four results are added.

Where the course uses it

0.3 What You Need, and How the Course Is Organized · Module 0, Start Here: What an LLM Doesfirst use
1.2 Matrix Multiplication, Done by Hand · Module 1, Shapes and Linear Algebra
1.4 Dot Products, Length, and Angle · Module 1, Shapes and Linear Algebra
2.3 The Chain Rule and the Jacobian · Module 2, Calculus and Optimization
3.2 Expectation and Variance · Module 3, Probability and Information
3.4 Maximum Likelihood · Module 3, Probability and Information
3.5 Information: Entropy, Cross-Entropy, KL, Perplexity · Module 3, Probability and Information
4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
5.6 The First Attention · Module 5, Sequences Before Transformers
6.3 Why Divide by the Square Root of the Key Width · Module 6, Attention
7.3 LayerNorm · Module 7, The Block and the Full Model
7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
8.2 Encoder-Only: BERT and Masked Language Modeling · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
8.5 Where the Norm Goes: Post-LN and Pre-LN · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
11.4 Tensor Parallelism · Module 11, Training Across Many GPUs
12.2 The KV Cache · Module 12, Generation
12.3 Decoding: From Logits to Text · Module 12, Generation
13.1 Perplexity and Its Limits · Module 13, Evaluation
14.1 The Residual Stream as Shared Memory · Module 14, What the Model Learns
15.3 RMSNorm · Module 15, The LLaMA Block
16.2 FlashAttention and the Online Softmax · Module 16, Efficient and Long Attention
17.2 Many Small Experts · Module 17, Mixture of Experts
17.4 Router Collapse and the Load-Balancing Loss · Module 17, Mixture of Experts
18.1 Linear Attention · Module 18, Beyond Attention: Linear Attention and Hybrids
20.1 Supervised Instruction Tuning · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
20.2 The Singular Value Decomposition · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
21.2 Policy Gradients · Module 21, Preference Learning: RL, RLHF and DPO
23.5 Speculative Decoding · Module 23, Serving
25.3 Contrastive Learning: CLIP and SigLIP · Module 25, Vision Transformers and Contrastive Learning
26.4 Training on Images and Text Together · Module 26, Vision-Language Models
27.3 The CTC Objective and Its Recursion · Module 27, Recognition: OCR and Speech With CTC
28.3 Vector Quantization and Residual VQ · Module 28, Generating Audio: Codecs and TTS

Related terms

∏ — productRead "the product of". The big pi works like ∑, but it multiplies the terms instead of adding them.
Dot product (inner product)Read "the dot product of u and v". Multiply the two vectors entry by entry and add the results. You get one number, which is large when the vectors point the same way.