# Module 28: Generating Audio: Codecs and TTS

> How speech becomes tokens a transformer can generate: spectrograms, autoencoders, vector quantization and RVQ codecs, Qwen3-TTS, GANs and diffusion.

- Part VII: Beyond Text · elective · Specialized · 3 hours
- By the end you can explain how speech becomes discrete tokens that a transformer can generate.
- Builds on: [Module 27: Recognition: OCR and Speech With CTC](https://transformers-and-math.drmhse.com/md/27-recognition-with-ctc.md), [Module 3: Probability and Information](https://transformers-and-math.drmhse.com/md/03-probability-and-information.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/28-generating-audio
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

Modern speech models generate audio as language models generate text, one token at a time, which needs a vocabulary of discrete tokens for sound. This module builds one: audio representations, autoencoders that compress a signal into a small learned representation, and vector quantization that turns it into integers. Then text-to-speech is next-token prediction, as section 28.4 shows for Qwen3-TTS.

## 28.1 Audio Basics

A microphone turns sound into a *waveform*: one air-pressure number per *sample*, at a *sample rate* of commonly 16,000 or 24,000 per second (16 or 24 kHz). Even a short sentence is far too long a sequence to model sample by sample with attention.

A listener hears mostly which *frequencies* are present, and how loud, over time. A *spectrogram* shows that: cut the waveform into overlapping windows (say 25 ms every 10 ms), measure each frequency's strength with a Fourier transform, and lay the results side by side, time across, frequency up, brightness for energy. A *mel spectrogram* spaces the frequency axis like human hearing, finely at low pitches and coarsely at high, usually in about 80 bands, turning 24,000 numbers per second into about 100 frames of 80.

> **Interactive figure** `waveform-to-mel`, on the web page: https://transformers-and-math.drmhse.com/modules/28-generating-audio#figure-waveform-to-mel
>
> **Try:** The signal is a low tone that jumps to a higher tone halfway through. Before looking at the spectrogram, predict what the bright band does at the jump, and where a click (a single sharp spike) would show up.
>
> **What happens:** The bright band steps up from a low row to a higher row at the jump: each tone is a horizontal line at its frequency. A click is the opposite: one moment, all frequencies, so it appears as a thin vertical line across the whole spectrogram.

> **In words.** A synthetic waveform above its spectrogram, time left to right. Presets: a steady tone, a rising sweep, two tones in sequence, and a tone with a click. Each frequency shows as a bright band at its height; a sudden event as a vertical stripe.

Older text-to-speech (TTS) systems turned text into a mel spectrogram, then a **vocoder** such as HiFi-GAN turned that into a waveform. Modern ones work like LLMs: a **neural audio codec** turns audio into discrete codes, as a tokenizer does text, a transformer predicts the codes, and the codec's decoder turns them back into sound.

## 28.2 Autoencoders and Latent Spaces

An *autoencoder* is trained to reproduce its input through a bottleneck:

- an **encoder** that maps the input $\mathbf{x}$ to a much smaller vector $\mathbf{z}$, the *latent* (or *code*);
- a **decoder** that maps $\mathbf{z}$ back to a reconstruction $\hat{\mathbf{x}}$.

Both train together to make $\hat{\mathbf{x}}$ close to $\mathbf{x}$, say by minimizing $\|\mathbf{x} - \hat{\mathbf{x}}\|^2$. The input is its own target, another kind of self-supervised learning (section 4.1). The latent is too small to copy into, so the network must learn a compact description keeping what matters; the space of latents is the *latent space*.

A neural audio codec is an autoencoder for sound. Its encoder, strided convolutions (section 25.1) over the waveform, gives one latent vector per short stretch, a few dozen per second; its decoder turns latents back into a waveform. Codecs add adversarial and perceptual losses (section 28.5), since a plain squared error sounds muffled. The latents are still continuous; to generate them as tokens they must become integers.

## 28.3 Vector Quantization and Residual VQ

*Vector quantization* (VQ) replaces a vector with the nearest entry of a learned list, the *codebook* (say 1024 vectors), storing only that entry's index $k$. It is section 24.2's quantization with learned points instead of an even grid. Entries are learned by moving each to the average of the vectors assigned to it (k-means), or by gradient descent with a straight-through estimator (section 24.5) through the nearest-neighbour step.

Describing audio finely with one codebook would need an enormous number of entries. *Residual vector quantization* uses several small codebooks in sequence instead.

```mermaid
graph TD
    ContinuousAudio["Raw waveform (24 kHz)"] --> AudioEncoder["Convolutional audio encoder"]
    AudioEncoder --> Latent["Continuous latent vector z (one per frame)"]
    Latent --> RVQ1["Codebook 1: nearest entry to z -> code k_1 (coarse)"]
    RVQ1 --> Res1["Residual r_1 = z - C_1[k_1]"]
    Res1 --> RVQ2["Codebook 2: nearest entry to r_1 -> code k_2"]
    RVQ2 --> Res2["Residual r_2 = r_1 - C_2[k_2]"]
    Res2 --> RVQ8["... up to codebook M: finest detail -> code k_M"]
```

> **In words.** The first codebook replaces each frame's vector with its closest entry; the leftover, the residual, is replaced by the second codebook's closest entry, and so on. Each level captures detail the earlier ones missed, and the chosen entries sum back to the vector.

For a latent $\mathbf{z} \in \mathbb{R}^D$ and codebooks $\mathcal{C}_m$:

1. Quantize with codebook $\mathcal{C}_1$: pick the index $k_1$ of the entry nearest to $\mathbf{z}$, so $\mathbf{e}_1 = \mathcal{C}_1[k_1]$. The residual is $\mathbf{r}_1 = \mathbf{z} - \mathbf{e}_1$.
2. Quantize the residual with codebook $\mathcal{C}_2$: $\mathbf{e}_2 = \mathcal{C}_2[k_2]$ is the entry nearest to $\mathbf{r}_1$, and the new residual is $\mathbf{r}_2 = \mathbf{r}_1 - \mathbf{e}_2$.
3. Repeat for all $M$ codebooks (typically $M = 8$ or $16$). The reconstruction is the sum of the chosen entries:

$$
\hat{\mathbf{z}} = \sum_{m=1}^M \mathcal{C}_m[k_m]
$$

A small example in two dimensions: let $\mathbf{z} = (0.9, -0.4)$. Codebook 1 holds $(1, 0)$ as its nearest entry, so $\mathbf{r}_1 = (-0.1, -0.4)$. Codebook 2's nearest entry to that is $(0, -0.5)$, so $\mathbf{r}_2 = (-0.1, 0.1)$. Two codes give $\hat{\mathbf{z}} = (1, -0.5)$, already close to $\mathbf{z}$, and a third codebook would shrink the leftover $(-0.1, 0.1)$ further.

> **Interactive figure** `rvq`, on the web page: https://transformers-and-math.drmhse.com/modules/28-generating-audio#figure-rvq
>
> **Try:** Before stepping, predict whether the error after the second codebook can be larger than after the first.
>
> **What happens:** Not if the codebook contains a zero vector, or any entry closer to the residual than zero is: the second stage can always choose to add something small. In practice each stage removes a share of what is left, so the error shrinks stage by stage and later codebooks carry finer and finer detail.

> **In words.** The example above: target 0.9, minus 0.4. The first entry, 1, 0, leaves an error of about 0.41; adding 0, minus 0.5 leaves about 0.14; adding minus 0.1, 0.1 lands exactly on z. The view zooms in at each stage as the leftover shrinks.

> **Browser lab** `rvq-stages`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/28-generating-audio#lab-rvq-stages

> **Not measured.** The reference repository has no RVQ or codec implementation; the browser lab above trains small codebooks on random vectors to check the mechanism, and the description of Qwen3-TTS below summarizes the published model card.

## 28.4 Qwen3-TTS: Predicting Codes

Each audio frame becomes $M$ integers. Qwen3-TTS's codec has 16 codebooks, and the "12Hz" in its model names is its frame rate.

Two transformers predict each frame's 16 codes. The main **talker** reads the text and voice conditioning (vectors describing the target speaker's voice) and predicts each frame's first code, frame by frame, as an LLM predicts tokens. A much smaller **depth predictor** fills in the other 15, codebook by codebook, each conditioned on those already chosen. The codec decoder turns the codes into a 24 kHz waveform, with no mel spectrogram or diffusion step anywhere.

## 28.5 Where GANs and Diffusion Fit

GANs and diffusion models are the main alternatives to audio tokens, and both appear inside token-based systems too.

- A **generative adversarial network** (GAN) trains a *generator* against a *discriminator* that tells its samples from real ones, until the generator's outputs are indistinguishable. GAN *vocoders* such as HiFi-GAN turn mel spectrograms into waveforms quickly and sharply, and codecs use a discriminator loss for the same reason: it penalizes the muffled, averaged sound a squared error accepts.
- A **diffusion model** learns to reverse gradual noising: training teaches a network to predict and remove noise added in small steps, and generation denoises step by step from pure noise. It dominates image generation and generates spectrograms or latents in some speech systems.

The token approach trades some of their strengths for one large advantage: once audio is a sequence of integers, everything in Modules 8 to 23 applies directly, from next-token training and scaling to KV caches and sampling.

## Exercises

**Exercise (apply).** How many integers does a 12 Hz codec with 16 codebooks produce for 10 seconds of audio, and how many raw samples does 10 seconds of 24 kHz audio have?

*Answer:* 1920 (within 0)

*Solution:* $10 \times 12 \times 16 = 1{,}920$ codes, against $10 \times 24{,}000 = 240{,}000$ samples: 125 times fewer numbers, and each code is a small integer.

**Exercise (apply).** Run two RVQ stages by hand on $\mathbf{z} = (2.2, 0.7)$ with codebook 1 $= \{(2, 0), (0, 2)\}$ and codebook 2 $= \{(0.25, 0.5), (-0.25, -0.5)\}$. What is the final residual?

*Solution:* Stage 1: $(2, 0)$ is nearer, leaving $\mathbf{r}_1 = (0.2, 0.7)$. Stage 2: $(0.25, 0.5)$ is nearer, leaving $\mathbf{r}_2 = (-0.05, 0.2)$. The reconstruction is $(2.25, 0.5)$.

**Exercise (check).** Why must audio latents be turned into integers before a transformer can generate them like text?

*Solution:* A language model outputs a probability distribution over a fixed vocabulary of discrete tokens and is trained with cross-entropy on the correct one. Continuous vectors have no such vocabulary; quantizing them into codebook indices gives every frame discrete tokens that the same next-token machinery can predict.

**Exercise (extend).** With 8 codebooks of 1,024 entries each, how many distinct reconstructions can RVQ represent per frame, and how large a single codebook would give the same number?

*Solution:* Each frame chooses one of 1,024 entries in each of 8 codebooks, so $1024^8 = 2^{80} \approx 1.2 \times 10^{24}$ combinations. A single codebook would need that many entries, impossible to store or search. RVQ gets the same expressiveness from $8 \times 1{,}024 = 8{,}192$ stored vectors.

### Core Academic References
- **Hinton & Salakhutdinov (2006)**: *Reducing the Dimensionality of Data with Neural Networks* (autoencoders). Science, 313(5786): 504–507.
- **van den Oord, Vinyals & Kavukcuoglu (2017)**: *Neural Discrete Representation Learning* (VQ-VAE). NeurIPS 2017. [arXiv:1711.00937](https://arxiv.org/abs/1711.00937)
- **Zeghidour et al. (2022)**: *SoundStream: An End-to-End Neural Audio Codec*. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495–507. [arXiv:2107.03312](https://arxiv.org/abs/2107.03312)
- **Défossez et al. (2022)**: *High Fidelity Neural Audio Compression* (EnCodec / RVQ). [arXiv:2210.13438](https://arxiv.org/abs/2210.13438)
- **Kong, Kim & Bae (2020)**: *HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis*. NeurIPS 2020. [arXiv:2010.05646](https://arxiv.org/abs/2010.05646)
- **Goodfellow et al. (2014)**: *Generative Adversarial Nets*. NeurIPS 2014. [arXiv:1406.2661](https://arxiv.org/abs/1406.2661)
- **Ho, Jain & Abbeel (2020)**: *Denoising Diffusion Probabilistic Models*. NeurIPS 2020. [arXiv:2006.11239](https://arxiv.org/abs/2006.11239)
- **Qwen Team (2025)**: *Qwen3-TTS* model card (Qwen3-TTS-12Hz-1.7B-Base). Hugging Face.