Modern speech models generate audio as language models generate text, one token at a time, which needs a vocabulary of discrete tokens for sound. This module builds one: audio representations, autoencoders that compress a signal into a small learned representation, and vector quantization that turns it into integers. Then text-to-speech is next-token prediction, as section 28.4 shows for Qwen3-TTS.

28.1 Audio BasicsRun this section

A microphone turns sound into a waveform: one air-pressure number per sample, at a sample rate of commonly 16,000 or 24,000 per second (16 or 24 kHz). Even a short sentence is far too long a sequence to model sample by sample with attention.

A listener hears mostly which frequencies are present, and how loud, over time. A spectrogram shows that: cut the waveform into overlapping windows (say 25 ms every 10 ms), measure each frequency's strength with a Fourier transform, and lay the results side by side, time across, frequency up, brightness for energy. A mel spectrogram spaces the frequency axis like human hearing, finely at low pitches and coarsely at high, usually in about 80 bands, turning 24,000 numbers per second into about 100 frames of 80.

Figure 28.1

From a waveform to a spectrogram

Top: the samples over one second. Bottom: the strength of each frequency (up) in each short window (across). Bright means strong.

A 4 kHz sample rate and a linear frequency axis, to keep the computation small; speech models use 16–24 kHz and a mel-spaced axis with about 80 bands.

Predict, then try

The signal is a low tone that jumps to a higher tone halfway through. Before looking at the spectrogram, predict what the bright band does at the jump, and where a click (a single sharp spike) would show up.

In words. A synthetic waveform above its spectrogram, time left to right. Presets: a steady tone, a rising sweep, two tones in sequence, and a tone with a click. Each frequency shows as a bright band at its height; a sudden event as a vertical stripe.

Older text-to-speech (TTS) systems turned text into a mel spectrogram, then a vocoder such as HiFi-GAN turned that into a waveform. Modern ones work like LLMs: a neural audio codec turns audio into discrete codes, as a tokenizer does text, a transformer predicts the codes, and the codec's decoder turns them back into sound.

28.2 Autoencoders and Latent SpacesRun this section

An is trained to reproduce its input through a bottleneck:

an encoder that maps the input to a much smaller vector $\mathbf{z}$ , the latent (or code);
a decoder that maps $\mathbf{z}$ back to a reconstruction .

Both train together to make $\hat{\mathbf{x}}$ close to $\mathbf{x}$ , say by minimizing . The input is its own target, another kind of self-supervised learning (section 4.1). The latent is too small to copy into, so the network must learn a compact description keeping what matters; the space of latents is the latent space.

A neural audio codec is an autoencoder for sound. Its encoder, strided convolutions (section 25.1) over the waveform, gives one latent vector per short stretch, a few dozen per second; its decoder turns latents back into a waveform. Codecs add adversarial and perceptual losses (section 28.5), since a plain squared error sounds muffled. The latents are still continuous; to generate them as tokens they must become integers.

28.3 Vector Quantization and Residual VQRun this section

Vector quantization (VQ) replaces a vector with the nearest entry of a learned list, the codebook (say 1024 vectors), storing only that entry's index $k$ . It is section 24.2's quantization with learned points instead of an even grid. Entries are learned by moving each to the average of the vectors assigned to it (k-means), or by gradient descent with a straight-through estimator (section 24.5) through the nearest-neighbour step.

Describing audio finely with one codebook would need an enormous number of entries. Residual vector quantization uses several small codebooks in sequence instead.

Architecture Flowchart

In words. The first codebook replaces each frame's vector with its closest entry; the leftover, the residual, is replaced by the second codebook's closest entry, and so on. Each level captures detail the earlier ones missed, and the chosen entries sum back to the vector.

For a latent and codebooks $\mathcal{C}_m$ :

Quantize with codebook $\mathcal{C}_1$ : pick the index $k_1$ of the entry nearest to $\mathbf{z}$ , so $\mathbf{e}_1 = \mathcal{C}_1[k_1]$ . The residual is $\mathbf{r}_1 = \mathbf{z} - \mathbf{e}_1$ .
Quantize the residual with codebook $\mathcal{C}_2$ : $\mathbf{e}_2 = \mathcal{C}_2[k_2]$ is the entry nearest to $\mathbf{r}_1$ , and the new residual is $\mathbf{r}_2 = \mathbf{r}_1 - \mathbf{e}_2$ .
Repeat for all $M$ codebooks (typically $M = 8$ or $16$ ). The reconstruction is the sum of the chosen entries:

\hat{\mathbf{z}} = \sum_{m=1}^M \mathcal{C}_m[k_m]

introduces 1 notation

A small example in two dimensions: let $\mathbf{z} = (0.9, -0.4)$ . Codebook 1 holds $(1, 0)$ as its nearest entry, so $\mathbf{r}_1 = (-0.1, -0.4)$ . Codebook 2's nearest entry to that is $(0, -0.5)$ , so $\mathbf{r}_2 = (-0.1, 0.1)$ . Two codes give $\hat{\mathbf{z}} = (1, -0.5)$ , already close to $\mathbf{z}$ , and a third codebook would shrink the leftover $(-0.1, 0.1)$ further.

Figure 28.3

Residual vector quantization, one codebook at a time

The target z is the section’s example, (0.9, −0.4). Each codebook offers nine entries around the running sum; the nearest one is chosen, and the view zooms in on what is left. Click anywhere to move z.

Four toy codebooks of nine entries each, spaced 1, 0.5, 0.1 and 0.04 apart. A real codec learns 1,024 entries per codebook in a vector with hundreds of dimensions.

Predict, then try

Before stepping, predict whether the error after the second codebook can be larger than after the first.

In words. The example above: target 0.9, minus 0.4. The first entry, 1, 0, leaves an error of about 0.41; adding 0, minus 0.5 leaves about 0.14; adding minus 0.1, 0.1 lands exactly on z. The view zooms in at each stage as the leftover shrinks.

Browser lab

Each residual codebook removes part of what is left

1,500 random 8-dimensional vectors stand in for codec latents. Six codebooks of 16 entries each are learned one after another by k-means, each on the residuals the previous ones left. Section 28.3 predicts the squared error falls with every stage, each stage adding 4 bits per vector.

Random vectors, not audio latents, and k-means instead of training jointly with a codec. It checks the residual mechanism, not the quality of any real codec.

Not measured. The reference repository has no RVQ or codec implementation; the browser lab above trains small codebooks on random vectors to check the mechanism, and the description of Qwen3-TTS below summarizes the published model card.

28.4 Qwen3-TTS: Predicting Codes

Each audio frame becomes $M$ integers. Qwen3-TTS's codec has 16 codebooks, and the "12Hz" in its model names is its frame rate.

Two transformers predict each frame's 16 codes. The main talker reads the text and voice conditioning (vectors describing the target speaker's voice) and predicts each frame's first code, frame by frame, as an LLM predicts tokens. A much smaller depth predictor fills in the other 15, codebook by codebook, each conditioned on those already chosen. The codec decoder turns the codes into a 24 kHz waveform, with no mel spectrogram or diffusion step anywhere.

28.5 Where GANs and Diffusion Fit

GANs and diffusion models are the main alternatives to audio tokens, and both appear inside token-based systems too.

A generative adversarial network (GAN) trains a generator against a discriminator that tells its samples from real ones, until the generator's outputs are indistinguishable. GAN vocoders such as HiFi-GAN turn mel spectrograms into waveforms quickly and sharply, and codecs use a discriminator loss for the same reason: it penalizes the muffled, averaged sound a squared error accepts.
A diffusion model learns to reverse gradual noising: training teaches a network to predict and remove noise added in small steps, and generation denoises step by step from pure noise. It dominates image generation and generates spectrograms or latents in some speech systems.

The token approach trades some of their strengths for one large advantage: once audio is a sequence of integers, everything in Modules 8 to 23 applies directly, from next-token training and scaling to KV caches and sampling.

Exercises

Apply

How many integers does a 12 Hz codec with 16 codebooks produce for 10 seconds of audio, and how many raw samples does 10 seconds of 24 kHz audio have?

Apply

Run two RVQ stages by hand on $\mathbf{z} = (2.2, 0.7)$ with codebook 1 $= \{(2, 0), (0, 2)\}$ and codebook 2 $= \{(0.25, 0.5), (-0.25, -0.5)\}$ . What is the final residual?

Check

Why must audio latents be turned into integers before a transformer can generate them like text?

Extend

With 8 codebooks of 1,024 entries each, how many distinct reconstructions can RVQ represent per frame, and how large a single codebook would give the same number?

Core Academic References

Hinton & Salakhutdinov (2006): Reducing the Dimensionality of Data with Neural Networks (autoencoders). Science, 313(5786): 504–507.
van den Oord, Vinyals & Kavukcuoglu (2017): Neural Discrete Representation Learning (VQ-VAE). NeurIPS 2017. arXiv:1711.00937
Zeghidour et al. (2022): SoundStream: An End-to-End Neural Audio Codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495–507. arXiv:2107.03312
Défossez et al. (2022): High Fidelity Neural Audio Compression (EnCodec / RVQ). arXiv:2210.13438
Kong, Kim & Bae (2020): HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. NeurIPS 2020. arXiv:2010.05646
Goodfellow et al. (2014): Generative Adversarial Nets. NeurIPS 2014. arXiv:1406.2661
Ho, Jain & Abbeel (2020): Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239
Qwen Team (2025): Qwen3-TTS model card (Qwen3-TTS-12Hz-1.7B-Base). Hugging Face.