ε — a tiny safety constant (or the CTC blank)
Read "epsilon". It is a very small number added to a denominator so the division can never be by zero. In CTC (section 27.2) the same letter means something else: the blank symbol, "no character here".
Typical values are or . In it is a safeguard: if all of a token's values happen to be equal, then and dividing by would give infinity or NaN ("not a number"). is small enough to have no visible effect normally and large enough to prevent that.
A second meaning. In CTC speech and text recognition (section 27.2), is not a number at all. It is the blank label a frame outputs when it adds no new character. It lets the model say "nothing here" and separates doubled letters, so c ε c a t reads as "ccat" while c c a t collapses to "cat".
Example
A token whose values are all equal becomes all zeros rather than breaking the computation.
Where the course uses it
- 7.3 LayerNorm · Module 7, The Block and the Full Modelfirst use
- 10.2 AdamW · Module 10, Pretraining
- 15.3 RMSNorm · Module 15, The LLaMA Block
- 19.2 A Config File, Field by Field · Module 19, Reading a Model Card
- 21.3 PPO · Module 21, Preference Learning: RL, RLHF and DPO
- 27.2 The Alignment Problem and the Blank · Module 27, Recognition: OCR and Speech With CTC
- 27.3 The CTC Objective and Its Recursion · Module 27, Recognition: OCR and Speech With CTC
Related terms
- LayerNorm — layer normalizationRead "layer norm". It shifts and rescales each token's vector to have mean 0 and variance 1, then applies a learned scale and shift.
- RMSNorm — root-mean-square normalizationRead "R M S norm". It is LayerNorm without the mean subtraction and without the shift (β): divide each vector by its root mean square, then apply a learned scale.