# Module 26: Vision-Language Models

> How vision-language models work: connectors from a ViT to an LLM, high-resolution tiling, 2D positions, and training on images and text together.

- Part VII: Beyond Text · elective · Specialized · 3 hours
- By the end you can build the input sequence of a VLM and train it correctly.
- Builds on: [Module 25: Vision Transformers and Contrastive Learning](https://transformers-and-math.drmhse.com/md/25-vision-transformers.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/26-vision-language-models
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

Many real inputs are images: photos, scanned documents, charts, screenshots, video frames. A **vision-language model (VLM)** takes images and text together and answers in text. The idea throughout: once an image is a list of vectors of the right width, a transformer treats it like any other sequence. This module connects a ViT (section 25.2) to a language model (26.1), handles large images without running out of tokens (26.2 and 26.3), and trains the combination (26.4).

## 26.1 Connecting Vision to Language

A VLM usually joins a separately trained vision encoder, often a CLIP or SigLIP ViT (section 25.3) whose vectors already relate to words, and a decoder-only language model such as LLaMA, Gemma or Qwen. The encoder's vectors have their own width $d_{\text{vis}}$ (not attention's value width $d_v$); the language model expects vectors of width $d_{\text{text}}$ that look like its token embeddings. A small trained **connector** converts one into the other.

```mermaid
graph LR
    ViT["Vision encoder (e.g. SigLIP or CLIP ViT)"] --> H_v["Image vectors H_v, N x d_vis"]
    H_v --> Connector["Connector (small trained network)"]
    Connector --> H_v_proj["Image tokens H_v', M x d_text"]
    H_v_proj --> LLM["Decoder-only language model"]
    TextTokens["Text tokens from the user's prompt"] --> LLM
```

> **In words.** The vision encoder turns the image into $N$ vectors; the connector turns them into $M$ vectors the language model reads like token embeddings, alongside the text prompt.

### Three Kinds of Connector

#### 1. Linear or 2-Layer MLP Projection (LLaVA Style)
The simplest connector maps each image vector on its own. LLaVA (Liu et al., 2023) used a single matrix; LLaVA-1.5 uses a 2-layer MLP with GELU:

$$
H_v' = \operatorname{GELU}(H_v W_1 + \mathbf{b}_1) W_2 + \mathbf{b}_2
$$

With $W_1 \in \mathbb{R}^{d_{\text{vis}} \times d_{\text{mid}}}$ and $W_2 \in \mathbb{R}^{d_{\text{mid}} \times d_{\text{text}}}$, each row changes width but the row count stays: $M = N$. It is simple and keeps every detail, but every patch takes a token position: $196$ to $576$ per image for common encoders.

#### 2. Perceiver Resampler / Q-Former (Flamingo & BLIP-2)
This connector outputs a fixed number of tokens however many patches go in. $M$ learned **queries** $\mathbf{q} \in \mathbb{R}^{M \times d}$ ($M = 64$ in Flamingo) cross-attend (section 8.1) to the image vectors, which supply the keys and values:

$$
H_v' = \operatorname{CrossAttention}(\mathbf{q}, H_v, H_v)
$$

Each query gives one output, so hundreds of patches become $M$ tokens. Real resamplers stack several such layers with feed-forward networks.

#### 3. Spatial Patch Merging / Pixel Unshuffle
A middle option: concatenate each $2 \times 2$ block of neighboring patch vectors into width $4 d_{\text{vis}}$ and map that to $d_{\text{text}}$. The count drops $75\%$ ($576 \to 144$), and the grid keeps its shape, coarser: $24 \times 24$ becomes $12 \times 12$.

> **Interactive figure** `connector-compare`, on the web page: https://transformers-and-math.drmhse.com/modules/26-vision-language-models#figure-connector-compare
>
> **Try:** With a 32 × 32 patch grid, predict which connectors still fit comfortably in a 4,096-token context alongside a long prompt.
>
> **What happens:** The MLP projection spends 1,024 tokens on a single image and patch merging 256; the resampler always spends 64. A plain projection keeps every detail but crowds the context, which is why high-resolution models merge or resample.

> **In words.** For a chosen patch grid, each connector's token count as a bar against a four-thousand-token context, with what each keeps or loses.

## 26.2 High Resolution: AnyRes Tiling

Documents, screenshots and diagrams need high resolution. A $2048 \times 2048$ image in $16 \times 16$ patches gives $16{,}384$ tokens, more than the whole 4,096-token context early VLMs such as LLaVA-1.5 were built on, and the ViT was trained at a fixed size such as $336 \times 336$. LLaVA-NeXT and InternVL use **tiling** (LLaVA-NeXT's **AnyRes**): encode a grid of tiles matching the image's shape at the ViT's own size, plus a shrunken copy of the whole image.

```mermaid
graph TD
    BigImg["High-resolution image, 1008 wide x 672 tall"] --> Global["Shrink the whole image to 336 x 336 <br/> (one overview tile)"]
    BigImg --> Grid["Cut into 2 rows x 3 columns <br/> of 336 x 336 tiles"]
    Global --> ViT1["Encode the overview with the ViT"]
    Grid --> ViT2["Encode each of the 6 tiles with the same ViT"]
    ViT1 --> Merge["Join all image tokens in order, <br/> with a separator token at the end of each row"]
    ViT2 --> Merge
    Merge --> LLM["Feed the joined image tokens to the language model"]
```

> **In words.** The shrunken overview tile gives the layout; the full-size tiles give the detail. All their tokens are joined in order, with separators at the end of each grid row.

Seven tiles cost seven times a single tile's tokens, which is why tiling is often combined with patch merging.

> **Interactive figure** `anyres-tiler`, on the web page: https://transformers-and-math.drmhse.com/modules/26-vision-language-models#figure-anyres-tiler
>
> **Try:** A wide 1008 × 336 image and a square 672 × 672 image hold the same number of pixels. Before trying them, predict which tile grid each gets and how many image tokens each costs at 576 tokens per tile, plus one overview tile.
>
> **What happens:** The wide image gets a 1 × 3 grid and the square one a 2 × 2 grid: 3 + 1 and 4 + 1 tiles, so 2,304 and 2,880 tokens. Tiling follows the image's shape, and every tile costs the same, so token cost tracks resolution closely. Patch merging divides each figure by four.

> **In words.** For an image size and tile size, the tile grid that best matches the image's shape, drawn with the overview tile beside it, and the tile and token counts with and without two-by-two patch merging.

## 26.3 2D RoPE for Image Patches

Numbered row by row, vertically touching patches end up a whole row apart: $14$ positions in a $14$-wide grid. **2D RoPE** gives each patch its row and column directly: for a patch at row $y$, column $x$, rotate the first half of the query (section 15.1) by $y$ and the second half by $x$:

$$
\mathbf{q}_{(y, x)} = \left[R_{\Theta_y, y} \, \mathbf{q}_{[:d/2]}, \quad R_{\Theta_x, x} \, \mathbf{q}_{[d/2:]}\right]
$$

Keys are rotated the same way, so in each half only the difference survives: scores depend on how many rows and columns apart two patches are. That directly signals neighbors, helping layout tasks such as reading tables or locating objects. Qwen2-VL's vision encoder uses it to read images at their own size instead of fixed tiles.

## 26.4 Training on Images and Text Together

### Merging Image and Text Into One Sequence
How `"User: What is this? <image>\nAssistant: It's a red fox."` becomes one sequence:
1. **Find the image marker**, a special token with its own ID (say `32000`).
2. **Turn each part into vectors.** The $T_1$ text tokens before it are looked up in $W_E$, giving $\mathbf{X}_{\text{prefix}} \in \mathbb{R}^{T_1 \times d}$; the image goes through the ViT and connector, giving $\mathbf{H}_v' \in \mathbb{R}^{N_{\text{patches}} \times d}$ (say 576 vectors, or the resampler's output count); the $T_2$ tokens after give $\mathbf{X}_{\text{suffix}} \in \mathbb{R}^{T_2 \times d}$.
3. **Stack them in order**, the image vectors replacing the marker:
   $$
   \mathbf{X}_{\text{unified}} = \left[\mathbf{X}_{\text{prefix}}; \, \mathbf{H}_v'; \, \mathbf{X}_{\text{suffix}}\right] \in \mathbb{R}^{(T_1 + N_{\text{patches}} + T_2) \times d}
   $$
   With $T_1 = 8$, $576$ image vectors and $T_2 = 10$, the language model sees $594$ positions and runs ordinary self-attention over all of them, image and text alike.

### Which Tokens Count Toward the Loss
**Visual instruction tuning** fine-tunes on image-question-answer examples with section 20.1's masked loss: **only the assistant's answer tokens count**.

$$
\mathcal{L}(\theta) = -\sum_{t=1}^T m_t \log P_\theta(w_t \mid \mathbf{X}_{\text{unified}}, w_{<t})
$$

The mask is
$$
m_t = \begin{cases} 1, & \text{if token } t \text{ is an assistant response token} \\ 0, & \text{if token } t \text{ is a user prompt, system prompt, or visual patch} \end{cases}
$$

Masked positions are still read as context. In PyTorch, set their labels to `-100`, the default `ignore_index` that `F.cross_entropy` skips (it also averages rather than sums, which only rescales the loss):
```python
# prompt and image positions stay in the input as context; -100 drops them from the loss
labels[prompt_mask] = -100
loss = F.cross_entropy(logits.view(-1, vocab_size), labels.view(-1), ignore_index=-100)
```
`prompt_mask` is true at every system-prompt, user-prompt and image position. Without it, the model would be trained to predict the question and to "predict" image positions, which are not vocabulary tokens at all.

> **Browser lab** `vlm-sequence`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/26-vision-language-models#lab-vlm-sequence

> **Interactive figure** `loss-mask-strip`, on the web page: https://transformers-and-math.drmhse.com/modules/26-vision-language-models#figure-loss-mask-strip
>
> **Try:** The sequence is a system prompt, a user question, 576 image positions and a short answer. Before switching the mask on, predict roughly what share of positions contribute to the loss.
>
> **What happens:** Only the answer tokens, which here are a few dozen out of more than six hundred positions: well under 10%. Everything else is context. Without the mask the loss would be dominated by trying to predict the prompt, and by "predicting" image positions that are not vocabulary tokens at all.

> **In words.** One training sequence as a strip of segments sized by length: system prompt, user text, image positions and answer. Switching the mask on greys out every position labelled minus 100, leaving only the answer, and the readout gives the scored share.

## Exercises

**Exercise (apply).** An AnyRes image uses a 2 × 3 grid of tiles plus one overview tile, each encoded to 576 tokens. How many image tokens is that, and how many after 2 × 2 patch merging?

*Answer:* 4032 (within 0)

*Solution:* $7 \times 576 = 4{,}032$ tokens; merging every 2 × 2 block divides by four, giving $1{,}008$.

**Exercise (apply).** A 24 × 24 grid of patch vectors goes through 2 × 2 patch merging. How many tokens come out, and what width is each merged vector before its projection, if $d_{\text{vis}} = 1024$?

*Answer:* 144 (within 0)

*Solution:* The grid becomes $12 \times 12 = 144$ tokens, and each merged vector joins four patch vectors end to end: $4 \times 1024 = 4096$ numbers before the projection to $d_{\text{text}}$.

**Exercise (check).** Why does a Perceiver resampler output the same number of tokens for every image, whatever its resolution?

*Solution:* Its outputs come from a fixed set of $M$ learned query vectors that cross-attend to the image patches. Each query produces exactly one output vector, so there are always $M$ outputs, however many patch vectors serve as keys and values.

**Exercise (extend).** With 2D RoPE, why does the attention score between two patches depend only on how many rows and columns apart they are?

*Solution:* The first half of each query and key is rotated by the row index and the second half by the column index. In each half the dot product of a rotated query and key depends only on the difference of their rotation angles (section 15.1), so the first half contributes a term depending only on the row difference and the second only on the column difference. The total score is the sum of the two.

### Core Academic References
- **Liu et al. (2023)**: *Visual Instruction Tuning* (LLaVA). NeurIPS 2023. [arXiv:2304.08485](https://arxiv.org/abs/2304.08485)
- **Liu et al. (2024)**: *Improved Baselines with Visual Instruction Tuning* (LLaVA-1.5). CVPR 2024. [arXiv:2310.03744](https://arxiv.org/abs/2310.03744)
- **Liu et al. (2024)**: *LLaVA-NeXT: Improved reasoning, OCR, and world knowledge*. Project blog post, January 2024.
- **Alayrac et al. (2022)**: *Flamingo: a Visual Language Model for Few-Shot Learning*. NeurIPS 2022. [arXiv:2204.14198](https://arxiv.org/abs/2204.14198)
- **Li et al. (2023)**: *BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models*. ICML 2023. [arXiv:2301.12597](https://arxiv.org/abs/2301.12597)
- **Chen et al. (2024)**: *InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks*. CVPR 2024. [arXiv:2312.14238](https://arxiv.org/abs/2312.14238)
- **Bai et al. (2023)**: *Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond*. [arXiv:2308.12966](https://arxiv.org/abs/2308.12966)
- **Wang et al. (2024)**: *Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution*. [arXiv:2409.12191](https://arxiv.org/abs/2409.12191)