# Module 19: Reading a Model Card

> Read a real model config field by field, compare GPT-2, LLaMA, Gemma and Qwen, and estimate parameter count, KV-cache size and compute.

- Part IV: The Modern Architecture · elective · Advanced · 2 hours
- By the end you can read a real model config and say what each field implies for memory, speed and quality.
- Builds on: [Module 15: The LLaMA Block](https://transformers-and-math.drmhse.com/md/15-the-llama-block.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/19-reading-a-model-card
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

An open model's card and configuration file answer most of what this course has taught you to ask: how wide, how deep, which attention, which position encoding, how big a vocabulary. This module compares the families you have met, then reads a configuration field by field and turns it into parameters, KV cache and compute per token.

## 19.1 Comparing the Architectures

The table collects the choices of Modules 15 to 17 for four families.

| Feature | GPT-2 (2019) | LLaMA-3 (2024) | Gemma 2 (2024) | Qwen2 / Qwen2.5 (2024) |
| :--- | :--- | :--- | :--- | :--- |
| **Normalization** | LayerNorm, before each sub-layer (Pre-LN) | RMSNorm, before each sub-layer | RMSNorm before and after each sub-layer | RMSNorm, before each sub-layer |
| **Position encoding** | Learned absolute positions | RoPE ($\theta = 500\text{k}$) | RoPE ($\theta = 10\text{k}$); Gemma 3: 10k local, 1M global | RoPE ($\theta = 1\text{M}$), YaRN (section 15.2) for long context |
| **Activation** | GELU | SwiGLU | GeGLU (gated GELU) | SwiGLU |
| **Attention** | Multi-head (MHA) | Grouped-query (GQA, 8 KV heads) | GQA, local and global layers alternating 1:1; Gemma 3: 5 local to 1 global | GQA |
| **Logit soft-capping** | None | None | Yes: $\kappa = 50$ in attention, $\kappa = 30$ on output logits (Gemma 3 uses QK-norm instead) | None |
| **Vocabulary size** | 50,257 tokens | 128,256 tokens | 256,000 tokens | 152,064 embedding rows |
| **Feed-forward** | Dense, 2 matrices | Dense SwiGLU, 3 matrices | Dense GeGLU, 3 matrices | Dense SwiGLU; MoE variant: 64 routed + 8 shared experts |
| **Model width $d_{\text{model}}$** | 768 (Small) | 4,096 (8B) | 3,584 (9B) | 3,584 (Qwen2.5-7B) |

*Every row is from the cited papers and model cards, not measured; none of these architectures is in the reference repository.*

> **Interactive figure** `architecture-explorer`, on the web page: https://transformers-and-math.drmhse.com/modules/19-reading-a-model-card#figure-architecture-explorer
>
> **Try:** Before opening it, guess which single feature changed the most between GPT-2 and the 2024 families: normalization, position encoding, feed-forward, or attention.
>
> **What happens:** Arguably position encoding: every 2024 family replaced GPT-2's learned table with RoPE, and then spent much of its effort stretching RoPE to longer contexts. The other rows changed too, but in smaller steps (LayerNorm to RMSNorm, GELU to a gated GELU or Swish, full heads to grouped heads).

> **In words.** The explorer lists architectures from the original Transformer and GPT-2 to LLaMA 3, Gemma 2, Qwen, a vision-language model, PP-OCR and Qwen3-TTS, and shows the chosen one's normalization, position encoding, activation, attention, vocabulary and loss beside the others.

## 19.2 A Config File, Field by Field

Models on Hugging Face ship a `config.json`. Here is one for an 8-billion-parameter LLaMA-3-style model:

```json
{
  "vocab_size": 128256,
  "hidden_size": 4096,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "intermediate_size": 14336,
  "hidden_act": "silu",
  "max_position_embeddings": 8192,
  "rope_theta": 500000.0,
  "rms_norm_eps": 1e-05,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16"
}
```

| field | meaning | section |
|---|---|---|
| `vocab_size` | $\lvert V\rvert$, rows of the embedding table | 9.4 |
| `hidden_size` | $d_{\text{model}}$, the width of the residual stream | 1.5, 7.6 |
| `num_hidden_layers` | the number of blocks $L$ | 7.6 |
| `num_attention_heads` | query heads $h_q$; the head width is `hidden_size / num_attention_heads` = 128 | 6.6 |
| `num_key_value_heads` | $h_{kv}$; fewer than $h_q$ means grouped-query attention, here 4 query heads per KV head | 15.5 |
| `intermediate_size` | $d_{ff}$, the inner width of the feed-forward network | 7.4, 15.4 |
| `hidden_act: silu` | Swish with $\beta = 1$; with three matrices this is SwiGLU | 15.4 |
| `max_position_embeddings` | the context length it was trained for | 15.2 |
| `rope_theta` | the RoPE base (10,000 in the original; raised for long context) | 15.1, 15.2 |
| `rms_norm_eps` | the $\epsilon$ inside RMSNorm | 15.3 |
| `tie_word_embeddings` | whether the output layer reuses the embedding table | 7.5 |
| `torch_dtype` | the format the weights are stored in, 2 bytes per number for bfloat16 | 1.7, 24.1 |

Other fields signal Modules 16 to 18: `sliding_window` or per-layer attention types for local and global layers, `num_experts` and `num_experts_per_tok` for mixture of experts, `rope_scaling` for context extension, and `final_logit_softcapping` for Gemma's soft cap.

## 19.3 Estimating Parameters, Cache, and Compute

With $d = $ `hidden_size`, $L = $ `num_hidden_layers`, $d_h = d / h_q$ and $d_{ff} = $ `intermediate_size`:

- **Parameters per block.** Attention has $W^Q$ and $W^O$ of size $d \times d$, and $W^K$ and $W^V$ of size $d \times h_{kv} d_h$, so $2d^2 + 2 d\,h_{kv} d_h$. A SwiGLU feed-forward network has three matrices, $3\,d\,d_{ff}$. Norms add a few times $d$.
- **Total parameters.** $L$ blocks plus the embedding table $\lvert V\rvert d$, plus the same again for the output layer if it is not tied.
- **KV cache per sequence** (sections 15.5 and 23.1): $2 \times L \times h_{kv} \times d_h \times (\text{context}) \times (\text{bytes})$.
- **Compute per token**, from section 10.5: about $2 \times$ (parameters) FLOPs in the forward pass, plus the attention scores, about $2 \times L \times d \times (\text{context})$ more, which matters only for long contexts.

> **Worked example.** For the configuration above: attention is $2 \cdot 4096^2 + 2 \cdot 4096 \cdot 8 \cdot 128 = 41.9$ million per block, SwiGLU is $3 \cdot 4096 \cdot 14{,}336 = 176.2$ million, so a block is about 218 million and 32 blocks about 6.98 billion. The untied embedding and output layer add $2 \cdot 128{,}256 \cdot 4096 = 1.05$ billion, for a total of about 8.03 billion: the "8B" in the name. At 8,192 tokens in bf16 the KV cache is $2 \cdot 32 \cdot 8 \cdot 128 \cdot 8192 \cdot 2$ bytes $= 1$ GiB per sequence.

> **Interactive figure** `config-reader`, on the web page: https://transformers-and-math.drmhse.com/modules/19-reading-a-model-card#figure-config-reader
>
> **Try:** Load the 8B configuration, then change `num_key_value_heads` from 8 to 32. Before you look, predict which of the three outputs change: parameters, KV cache, or compute per token.
>
> **What happens:** All three, but very unequally. The KV cache grows fourfold, from 1 GiB to 4 GiB at 8,192 tokens. Parameters and compute grow by only about 0.8 billion, the larger $W^K$ and $W^V$ across 32 layers, roughly 10%. Grouped-query attention is almost free in parameters and very valuable in memory.

> **In words.** An editable model configuration, with presets, and a readout of head width, parameters per block and in total, KV cache at a chosen context, and forward compute per token.

## Exercises

**Exercise (check).** Which configuration field tells you whether a model uses grouped-query attention?

*Solution:* `num_key_value_heads`: if it is smaller than `num_attention_heads`, several query heads share each key-value head.

**Exercise (apply).** A configuration has `hidden_size` 2048, `num_attention_heads` 16, `num_key_value_heads` 4. What is the head width, and how many query heads share each key-value head?

*Solution:* The head width is $2048 / 16 = 128$, and each key-value head serves $16 / 4 = 4$ query heads.

**Exercise (apply).** Estimate the KV cache in GiB for one 32,768-token sequence of a model with 28 layers, 4 key-value heads, head width 128, in bf16.

*Answer:* 1.75 (within 0.01)

*Solution:* $2 \times 28 \times 4 \times 128 \times 32{,}768 \times 2 = 1{,}879{,}048{,}192$ bytes, which is $1.75$ GiB.

**Exercise (apply).** How many parameters does one SwiGLU feed-forward network have at `hidden_size` 4096 and `intermediate_size` 14336, in millions?

*Answer:* 176.16 (within 0.1)

*Solution:* Three matrices of $4096 \times 14{,}336$: $3 \times 58{,}720{,}256 = 176{,}160{,}768$, about 176 million.

**Exercise (extend).** A configuration has `num_experts` 128 and `num_experts_per_tok` 8. How do you expect its total parameter count and its compute per token to compare with a dense model of the same width and depth whose feed-forward network is the size of one expert?

*Solution:* Total parameters are far larger: 128 expert feed-forward networks per layer instead of one, so the feed-forward parameters are about 128 times those of the dense model. Compute per token is only about 8 times the dense model's feed-forward compute, since each token runs 8 experts (plus a small router), and attention is unchanged. That gap between total and active parameters is the point of mixture of experts.

### Core Academic References
- **Radford et al., OpenAI (2019)**: *Language Models are Unsupervised Multitask Learners* (GPT-2). Technical Report.
- **Grattafiori et al. (2024)**: *The Llama 3 Herd of Models*. [arXiv:2407.21783](https://arxiv.org/abs/2407.21783)
- **Gemma Team, Google DeepMind (2024)**: *Gemma 2: Improving Open Language Models at a Practical Size*. [arXiv:2408.00118](https://arxiv.org/abs/2408.00118)
- **Qwen Team, Alibaba (2024)**: *Qwen2.5 Technical Report*. [arXiv:2412.15115](https://arxiv.org/abs/2412.15115)
- **Mitchell et al. (2019)**: *Model Cards for Model Reporting*. FAT* 2019. [arXiv:1810.03993](https://arxiv.org/abs/1810.03993)