Distillation — a student imitating a teacher
Training a smaller student model to match a larger teacher's output distribution, or to imitate text the teacher generated.
The teacher's full distribution carries more information than a one-hot label (section 20.4).