Vision-Language Models

Many real inputs are images: photos, scanned documents, charts, screenshots, video frames. A vision-language model (VLM) takes images and text together and answers in text. The idea throughout: once an image is a list of vectors of the right width, a transformer treats it like any other sequence. This module connects a ViT (section 25.2) to a language model (26.1), handles large images without running out of tokens (26.2 and 26.3), and trains the combination (26.4).

26.1 Connecting Vision to LanguageRun this section

A VLM usually joins a separately trained vision encoder, often a CLIP or SigLIP ViT (section 25.3) whose vectors already relate to words, and a decoder-only language model such as LLaMA, Gemma or Qwen. The encoder's vectors have their own width $d_{\text{vis}}$ (not attention's value width $d_v$); the language model expects vectors of width $d_{\text{text}}$ that look like its token embeddings. A small trained connector converts one into the other.

In words. The vision encoder turns the image into $N$ vectors; the connector turns them into $M$ vectors the language model reads like token embeddings, alongside the text prompt.

Three Kinds of Connector

1. Linear or 2-Layer MLP Projection (LLaVA Style)

The simplest connector maps each image vector on its own. LLaVA (Liu et al., 2023) used a single matrix; LLaVA-1.5 uses a 2-layer MLP with GELU:

With $W_1 \in \mathbb{R}^{d_{\text{vis}} \times d_{\text{mid}}}$ and $W_2 \in \mathbb{R}^{d_{\text{mid}} \times d_{\text{text}}}$, each row changes width but the row count stays: $M = N$. It is simple and keeps every detail, but every patch takes a token position: $196$ to $576$ per image for common encoders.

2. Perceiver Resampler / Q-Former (Flamingo & BLIP-2)

This connector outputs a fixed number of tokens however many patches go in. $M$ learned queries $\mathbf{q} \in \mathbb{R}^{M \times d}$ ($M = 64$ in Flamingo) cross-attend (section 8.1) to the image vectors, which supply the keys and values:

Each query gives one output, so hundreds of patches become $M$ tokens. Real resamplers stack several such layers with feed-forward networks.

3. Spatial Patch Merging / Pixel Unshuffle

A middle option: concatenate each $2 \times 2$ block of neighboring patch vectors into width $4 d_{\text{vis}}$ and map that to $d_{\text{text}}$. The count drops $75\%$ ($576 \to 144$), and the grid keeps its shape, coarser: $24 \times 24$ becomes $12 \times 12$.

In words. For a chosen patch grid, each connector's token count as a bar against a four-thousand-token context, with what each keeps or loses.

26.2 High Resolution: AnyRes TilingRun this section

Documents, screenshots and diagrams need high resolution. A $2048 \times 2048$ image in $16 \times 16$ patches gives $16{,}384$ tokens, more than the whole 4,096-token context early VLMs such as LLaVA-1.5 were built on, and the ViT was trained at a fixed size such as $336 \times 336$. LLaVA-NeXT and InternVL use tiling (LLaVA-NeXT's AnyRes): encode a grid of tiles matching the image's shape at the ViT's own size, plus a shrunken copy of the whole image.

In words. The shrunken overview tile gives the layout; the full-size tiles give the detail. All their tokens are joined in order, with separators at the end of each grid row.

Seven tiles cost seven times a single tile's tokens, which is why tiling is often combined with patch merging.

In words. For an image size and tile size, the tile grid that best matches the image's shape, drawn with the overview tile beside it, and the tile and token counts with and without two-by-two patch merging.

26.3 2D RoPE for Image Patches

Numbered row by row, vertically touching patches end up a whole row apart: $14$ positions in a $14$-wide grid. 2D RoPE gives each patch its row and column directly: for a patch at row $y$, column $x$, rotate the first half of the query (section 15.1) by $y$ and the second half by $x$:

Keys are rotated the same way, so in each half only the difference survives: scores depend on how many rows and columns apart two patches are. That directly signals neighbors, helping layout tasks such as reading tables or locating objects. Qwen2-VL's vision encoder uses it to read images at their own size instead of fixed tiles.

26.4 Training on Images and Text TogetherRun this section

Merging Image and Text Into One Sequence

How "User: What is this? <image>\nAssistant: It's a red fox." becomes one sequence:

1. Find the image marker, a special token with its own ID (say 32000).
2. Turn each part into vectors. The $T_1$ text tokens before it are looked up in $W_E$, giving $\mathbf{X}_{\text{prefix}} \in \mathbb{R}^{T_1 \times d}$; the image goes through the ViT and connector, giving $\mathbf{H}_v' \in \mathbb{R}^{N_{\text{patches}} \times d}$ (say 576 vectors, or the resampler's output count); the $T_2$ tokens after give $\mathbf{X}_{\text{suffix}} \in \mathbb{R}^{T_2 \times d}$.
3. Stack them in order, the image vectors replacing the marker:
   $$\mathbf{X}_{\text{unified}} = \left[\mathbf{X}_{\text{prefix}}; \, \mathbf{H}_v'; \, \mathbf{X}_{\text{suffix}}\right] \in \mathbb{R}^{(T_1 + N_{\text{patches}} + T_2) \times d}$$

   What does this say?
   With $T_1 = 8$, $576$ image vectors and $T_2 = 10$, the language model sees $594$ positions and runs ordinary self-attention over all of them, image and text alike.

Which Tokens Count Toward the Loss

Visual instruction tuning fine-tunes on image-question-answer examples with section 20.1's masked loss: only the assistant's answer tokens count.

The mask is

Masked positions are still read as context. In PyTorch, set their labels to -100, the default ignore_index that F.cross_entropy skips (it also averages rather than sums, which only rescales the loss):

# prompt and image positions stay in the input as context; -100 drops them from the loss
labels[prompt_mask] = -100
loss = F.cross_entropy(logits.view(-1, vocab_size), labels.view(-1), ignore_index=-100)

prompt_mask is true at every system-prompt, user-prompt and image position. Without it, the model would be trained to predict the question and to "predict" image positions, which are not vocabulary tokens at all.

In words. One training sequence as a strip of segments sized by length: system prompt, user text, image positions and answer. Switching the mask on greys out every position labelled minus 100, leaving only the answer, and the readout gives the scored share.

Exercises

Core Academic References

• Liu et al. (2023): Visual Instruction Tuning (LLaVA). NeurIPS 2023. arXiv:2304.08485
• Liu et al. (2024): Improved Baselines with Visual Instruction Tuning (LLaVA-1.5). CVPR 2024. arXiv:2310.03744
• Liu et al. (2024): LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. Project blog post, January 2024.
• Alayrac et al. (2022): Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022. arXiv:2204.14198
• Li et al. (2023): BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML 2023. arXiv:2301.12597
• Chen et al. (2024): InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. CVPR 2024. arXiv:2312.14238
• Bai et al. (2023): Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966
• Wang et al. (2024): Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv:2409.12191