# Module 25: Vision Transformers and Contrastive Learning
> Images as tensors, convolutions, the Vision Transformer (ViT), contrastive learning with CLIP and SigLIP, and zero-shot image classification.
- Part VII: Beyond Text · elective · Specialized · 4 hours
- By the end you can turn an image into tokens, and explain how image and text encoders are aligned.
- Builds on: [Module 8: Three Families: Encoder, Encoder-Decoder, Decoder](https://transformers-and-math.drmhse.com/md/08-three-families.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/25-vision-transformers
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com
This module shows how the same transformer reads images, and how image and text models are trained to understand each other: from convolutional networks to the Vision Transformer, which treats an image as a sequence of patches, and contrastive learning, which aligns image and text encoders using only pictures and their captions. Module 26 uses these encoders to give language models sight.
## 25.1 Images, Convolutions, and Inductive Bias
A colour image is an $H \times W \times C$ tensor (section 1.1), with $C = 3$ channels (red, green, blue) of brightnesses, often scaled to 0 to 1. A 224 by 224 image holds $150{,}528$ numbers. Fed as one long vector to an MLP, the first layer alone would need hundreds of millions of weights, and would have to learn separately that a cat's ear looks the same in every corner.
### Convolutions
A *convolution* fixes both. It slides a small grid of weights, a *filter* or *kernel* (say 3 by 3), across the image, writing the dot product with each patch underneath into a *feature map*. A handful of weights, reused everywhere, detects the same pattern, such as a vertical edge, wherever it occurs. A layer has many filters, and the next layer's filters combine their maps: edges into textures, textures into parts, parts into objects.
> **Interactive figure** `conv-filter`, on the web page: https://transformers-and-math.drmhse.com/modules/25-vision-transformers#figure-conv-filter
>
> **Try:** Choose the vertical-edge filter, whose left column is negative and right column positive. Before applying it, predict where the output feature map will be bright on an image of a white square on a black background.
>
> **What happens:** Along the square's left and right sides, with opposite signs: the filter responds where brightness changes from left to right, and gives zero on flat regions and on horizontal edges. The same nine weights find the edge wherever it is.
> **In words.** A small black-and-white image, an editable 3 by 3 filter (presets: vertical edge, horizontal edge, blur, sharpen) and the resulting feature map. Hovering an output cell highlights its pixel patch and shows the dot product.
A *convolutional neural network* (CNN) stacks convolutions, usually with pooling layers that shrink the maps. It builds in two assumptions: nearby pixels matter most, and a pattern means the same anywhere. Such built-in assumptions are *inductive biases*; they let CNNs learn from relatively little data, and CNNs dominated vision for a decade before transformers.
## 25.2 The Vision Transformer
The **Vision Transformer (ViT)** (Dosovitskiy et al., 2020) is the ordinary transformer of Module 7 with one new first step, which turns an image into a sequence.
```mermaid
graph TD
Img["Image, H x W x C
e.g. 224 x 224 x 3 (RGB)"] --> Patches["Cut into non-overlapping P x P patches
P = 16 gives (224/16)^2 = 196 patches"]
Patches --> Flatten["Flatten each patch into one vector
16 x 16 x 3 = 768 numbers"]
Flatten --> Linear["Multiply by the same matrix E
768 -> d_model"]
Linear --> Prepend["Put a learned [CLS] vector in front
and add a position vector to each token"]
Prepend --> Enc["Ordinary transformer encoder layers"]
Enc --> Out["Output sequence z_0, z_1, ..., z_N
one vector per patch, plus [CLS]"]
```
> **In words.** Cut the image into small square patches, flatten each, multiply by one shared matrix to the model's width, add a position vector, and run a normal transformer encoder. Each patch is one token.
### The Patch Formulas
For an image $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$:
1. Cut it into non-overlapping $P \times P$ patches, image area over patch area:
$$
N = \frac{HW}{P^2} \quad \text{patches}
$$
For $224 \times 224$ and $P = 16$, $N = 196$, a $14 \times 14$ grid.
2. Flatten each patch into $\mathbf{x}_p^{(i)} \in \mathbb{R}^{P^2 C}$: $16 \cdot 16 \cdot 3 = 768$ numbers.
3. Multiply each by the same learned matrix $E$ to width $d$, put a learned **class token** `[CLS]` in front, and add learned position vectors:
$$
\mathbf{z}_0 = \left[\mathbf{x}_{\text{class}}; \, \mathbf{x}_p^{(1)}E; \, \dots; \, \mathbf{x}_p^{(N)}E\right] + E_{pos}
$$
$E \in \mathbb{R}^{(P^2 C) \times d}$, and the semicolons stack rows, so $\mathbf{z}_0$ and $E_{pos} \in \mathbb{R}^{(N+1) \times d}$ have one row for the class token and one per patch. The encoder's final class-token vector summarizes the image, for classification, say.
> **Interactive figure** `vit-patches`, on the web page: https://transformers-and-math.drmhse.com/modules/25-vision-transformers#figure-vit-patches
>
> **Try:** A 224 × 224 image with 16 × 16 patches gives 196 tokens. Before changing the patch size, predict the token count with 8 × 8 patches, and what that does to the cost of attention.
>
> **What happens:** $(224/8)^2 = 784$ tokens, four times as many. Attention's score table grows with the square of the token count, so it costs sixteen times as much. Patch size trades detail against cost.
> **In words.** A small image cut into a four-by-four grid of 16-pixel patches. Each patch's red, green and blue planes of 256 numbers are laid end to end into 768 numbers, and multiplying by E makes one token. The encoder receives the class token, then one token per patch in reading order.
Multiplying by $E$ plays the role of section 1.5's embedding lookup. Beyond the patching, ViT has far fewer inductive biases than a CNN: its positions are learned and every patch attends to every other from the first layer, so it must learn from data which patches relate. The ViT paper found this pays off with a lot of data, while CNNs do better on smaller datasets.
> **Measured** (lab 6, `experiments/vit_patchify.py`). The claim that ViT is "the standard transformer with almost no changes" can be tested, and it holds. Cutting a $224 \times 224$ RGB image into $P = 16$ patches gives a $14 \times 14$ grid of vectors, each $16^2 \times 3 = 768$ numbers long: $196$ tokens. Adding a `[CLS]` token makes $197$. Passing those $197$ vectors through `src/transformer/transformer_block.py`, **with no changes to that file**, gives an output of shape $(1, 197, 64)$: one image, $197$ tokens, each $64$ wide. The block cannot tell whether its input came from text or pixels. The only thing it requires of its input is the vector width, `d_model`.
## 25.3 Contrastive Learning: CLIP and SigLIP
Classification needs labelled images, which limits both the data's scale and what is learned (a fixed list of classes). CLIP (Radford et al., 2021) instead learns from the web's billions of captioned images, with no labelling.
It trains a ViT for images and a transformer for text, each producing one vector per input, projected to a shared width and normalized to length 1. For a batch of $N$ image-caption pairs, all $N \times N$ cosine similarities (section 1.4), scaled by a learned temperature $\tau$, form a matrix with the true pairs on the diagonal. Softmax cross-entropy over the captions for each image, and over the images for each caption, averaged, gives the *InfoNCE* (contrastive) loss:
$$
\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^{N} \left[ \log \frac{e^{s_{ii}/\tau}}{\sum_{j} e^{s_{ij}/\tau}} + \log \frac{e^{s_{ii}/\tau}}{\sum_{j} e^{s_{ji}/\tau}} \right]
$$
where $s_{ij}$ is the similarity of image $i$ and caption $j$. Each true pair is pulled together and pushed from the other $N - 1$, which serve as free *negative examples*; bigger batches give more negatives and better models, and CLIP used 32,768.
SigLIP (Zhai et al., 2023) replaces the softmax with an independent sigmoid (section 3.6) on each entry, labelled 1 on the diagonal and 0 elsewhere. With no normalizing across entries, very large batches split across devices more easily, and SigLIP encoders are now common in vision-language models.
> **Interactive figure** `contrastive-matrix`, on the web page: https://transformers-and-math.drmhse.com/modules/25-vision-transformers#figure-contrastive-matrix
>
> **Try:** Before training starts, the similarity matrix is noise. Predict what it should look like after training, and what the loss would be if every image matched its own caption perfectly and no other.
>
> **What happens:** A bright diagonal on a dark background: each image is most similar to its own caption. With perfect matching and a low temperature, every softmax puts nearly all its weight on the diagonal and the loss approaches zero.
> **In words.** Image and caption embeddings as two sets of arrows, beside their similarity matrix. Training on the contrastive loss rotates matching arrows together, brightens the diagonal and lowers the loss.
## 25.4 Zero-Shot Classification
A trained CLIP model can classify images into categories it never trained on: *zero-shot classification*.
1. Write one caption per candidate class, from a template: "a photo of a dog", "a photo of a cat", "a photo of a car".
2. Encode the captions with the text encoder and the image with the image encoder.
3. Predict the class whose caption is most similar to the image.
Classes are chosen at use time in plain language. On ImageNet, CLIP's zero-shot accuracy matched a ResNet-50 trained on its 1.28 million labelled images, without seeing any (Radford et al., 2021). Template wording matters, as prompt wording does (section 13.2), and averaging several templates helps.
More important here, contrastive training yields an image encoder whose vectors already line up with the meaning of words, the natural front end for a vision-language model (section 26.1).
## Exercises
**Exercise (apply).** How many patches does a 336 × 336 image give with 14 × 14 patches, and how many numbers is each flattened patch?
*Answer:* 576 (within 0)
*Solution:* $(336/14)^2 = 24^2 = 576$ patches, each $14 \times 14 \times 3 = 588$ numbers.
**Exercise (apply).** A 3 × 3 filter with all weights $\tfrac19$ is applied to a patch of pixels whose values are 0 except for a single 9 in the centre. What is the output at that position?
*Answer:* 1 (within 0)
*Solution:* The dot product is $\tfrac19 \times 9 = 1$. This filter averages its 3 × 3 neighbourhood: a blur.
**Exercise (apply).** In a batch of 3 image-caption pairs with temperature $\tau = 1$, image 1's similarities to the three captions are $[2, 0, 0]$. What is its image-to-text loss term, $-\log$ of the softmax weight on its own caption?
*Answer:* 0.2395 (within 0.001)
*Solution:* $\frac{e^2}{e^2 + 1 + 1} = \frac{7.389}{9.389} = 0.787$, so the term is $-\log 0.787 \approx 0.24$.
**Exercise (check).** Why do the other pairs in a CLIP batch count as negative examples, and why does a larger batch help?
*Solution:* Each caption in the batch belongs to a different image, so for image $i$ every caption $j \ne i$ is a caption it should *not* match. The loss pushes image $i$ away from all of them. A larger batch supplies more, and more varied, negatives, including hard ones that are similar to the true caption, which forces finer distinctions.
**Exercise (extend).** Describe how to classify an image into 1,000 categories with a CLIP model and no training, and name one way the result depends on choices other than the model.
*Solution:* Write a caption for each category from a template ("a photo of a {class}"), encode all 1,000 captions and the image, and pick the category whose caption embedding has the highest cosine similarity with the image embedding. The result depends on the template wording and on how the class names are phrased (synonyms, ambiguous names), which is why ensembles of templates are used.
### Core Academic References
- **LeCun et al. (1998)**: *Gradient-Based Learning Applied to Document Recognition*. Proceedings of the IEEE, 86(11): 2278–2324.
- **He et al. (2016)**: *Deep Residual Learning for Image Recognition*. CVPR 2016: 770–778.
- **Dosovitskiy et al. (2020)**: *An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale* (ViT). ICLR 2021. [arXiv:2010.11929](https://arxiv.org/abs/2010.11929)
- **Oord, Li & Vinyals (2018)**: *Representation Learning with Contrastive Predictive Coding* (InfoNCE). [arXiv:1807.03748](https://arxiv.org/abs/1807.03748)
- **Radford et al. (2021)**: *Learning Transferable Visual Models From Natural Language Supervision* (CLIP). ICML 2021. [arXiv:2103.00020](https://arxiv.org/abs/2103.00020)
- **Zhai et al. (2023)**: *Sigmoid Loss for Language Image Pre-Training* (SigLIP). ICCV 2023. [arXiv:2303.15343](https://arxiv.org/abs/2303.15343)