e^{z}

The exponential function

Read "e to the z" (also written exp(z)). It raises e ≈ 2.718 to the power z. The result is always positive: e⁰ = 1, and it shrinks toward 0 as z becomes very negative.

Two facts explain how the course uses it:

$e^{z} > 0$ for every real $z$ . So softmax can never output a negative probability.
$e^{a+b} = e^a e^b$ . So subtracting the same constant from every input of a softmax multiplies the top and bottom of the fraction by the same factor, and it cancels exactly.

As $z$ goes to $-\infty$ , $e^{z}$ goes to $0$ . That is how the causal mask works: a score of $-\infty$ becomes a weight of exactly zero.

Example

e^{0} = 1, \quad e^{1} \approx 2.718, \quad e^{-5} \approx 0.0067

Positive in every case; large negative inputs give tiny positive outputs.

Where the course uses it

0.3 What You Need, and How the Course Is Organized · Module 0, Start Here: What an LLM Doesfirst use
3.1 Random Variables, Distributions, and Sampling · Module 3, Probability and Information
3.5 Information: Entropy, Cross-Entropy, KL, Perplexity · Module 3, Probability and Information
3.6 The Sigmoid and Preferences Between Pairs · Module 3, Probability and Information
4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
6.3 Why Divide by the Square Root of the Key Width · Module 6, Attention
8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
15.4 SwiGLU · Module 15, The LLaMA Block
16.2 FlashAttention and the Online Softmax · Module 16, Efficient and Long Attention
21.6 Direct Preference Optimization · Module 21, Preference Learning: RL, RLHF and DPO
24.1 How Floating-Point Numbers Are Stored · Module 24, Quantization and Number Formats
25.3 Contrastive Learning: CLIP and SigLIP · Module 25, Vision Transformers and Contrastive Learning

Related terms

SoftmaxRead "softmax of z". It turns any list of numbers into probabilities: every output is positive and they add up to 1. Bigger inputs get bigger shares.
LogarithmRead "log of x". It undoes the exponential: log(eᶻ) = z. It turns products into sums, which is why loss functions use it.
M — the attention maskA matrix of 0s and −∞ (minus infinity) added to the attention scores before softmax. A 0 leaves a position allowed; −∞ blocks it.