Transformers & Math

Course map

Thirty modules in eight parts. The core track (Modules 0–15, 21–23 and the capstone) is the path every learner follows; electives go deeper and can be taken once the modules they build on are done. Hover a module to highlight what it builds on (gold) and what builds on it (green).

Orientation

Module 00Core
Start Here: What an LLM Does
45 Minutes

Part I · Foundations

Module 01Core
Shapes and Linear Algebra
5 Hours · builds on 0
Module 02Core
Calculus and Optimization
4 Hours · builds on 1
Module 03Core
Probability and Information
4 Hours · builds on 2
Module 04Core
How a Model Learns From Data
5 Hours · builds on 3
Module 05Core
Sequences Before Transformers
3 Hours · builds on 4

Part II · The Transformer

Module 06Core
Attention
5 Hours · builds on 5
Module 07Core
The Block and the Full Model
5 Hours · builds on 6
Module 08Core
Three Families: Encoder, Encoder-Decoder, Decoder
4 Hours · builds on 7

Part III · Building an LLM

Module 09Core
Tokenization
3 Hours · builds on 8
Module 10Core
Pretraining
5 Hours · builds on 9
Module 11Core
Training Across Many GPUs
4 Hours · builds on 10
Module 12Core
Generation
3 Hours · builds on 8
Module 13Core
Evaluation
3 Hours · builds on 8, 3
Module 14Core
What the Model Learns
4 Hours · builds on 12

Part IV · The Modern Architecture

Module 15Core
The LLaMA Block
5 Hours · builds on 7, 10
Module 16Elective
Efficient and Long Attention
4 Hours · builds on 15
Module 17Elective
Mixture of Experts
3 Hours · builds on 15
Module 18Elective
Beyond Attention: Linear Attention and Hybrids
3 Hours · builds on 15, 5
Module 19Elective
Reading a Model Card
2 Hours · builds on 15

Part V · From Model to Assistant

Module 20Elective
Adaptation: Fine-Tuning, LoRA, Distillation
4 Hours · builds on 10, 12
Module 21Core
Preference Learning: RL, RLHF and DPO
5 Hours · builds on 12, 3
Module 22Core
Reasoning Models
4 Hours · builds on 21, 13

Part VI · Deployment

Module 23Core
Serving
4 Hours · builds on 12, 15
Module 24Elective
Quantization and Number Formats
4 Hours · builds on 23

Part VII · Beyond Text

Module 25Elective
Vision Transformers and Contrastive Learning
4 Hours · builds on 8
Module 26Elective
Vision-Language Models
3 Hours · builds on 25
Module 27Elective
Recognition: OCR and Speech With CTC
3 Hours · builds on 4, 6
Module 28Elective
Generating Audio: Codecs and TTS
3 Hours · builds on 27, 3

Part VIII · Capstone

Module 29Core
Capstone: Build, Upgrade, Measure, Ship a Tiny GPT
10 Hours · builds on 15, 23