\operatorname{TopK}(\mathcal{G}(\mathbf{x}))

Mixture-of-experts routing

A small gate network scores every expert for a token, only the K highest-scoring experts run, and their outputs are combined using the gate's weights.

$\mathcal{G}(\mathbf{x})$ (curly G, the gate) gives one score per expert. $\operatorname{TopK}$ keeps the $K$ best, for example 8 of 64 in a fine-grained design, or 2 of 16 in a coarser one. $g_i(\mathbf{x})$ is the weight for chosen expert $i$ 's output. $N_E$ is the total number of experts. The parameter count grows with the number of experts, but the compute per token grows only with $K$ .

The difficulty is balance. Left alone, the gate tends to send most tokens to a few favorite experts. The auxiliary loss $\mathcal{L}_{\text{aux}}$ sums $f_i P_i$ over experts, where $f_i$ is the fraction of tokens sent to expert $i$ and $P_i$ its average gate probability. That sum is smallest when tokens are spread evenly.

Where the course uses it

17.2 Many Small Experts · Module 17, Mixture of Expertsfirst use
17.4 Router Collapse and the Load-Balancing Loss · Module 17, Mixture of Experts
17.5 Expert Capacity and Dropped Tokens · Module 17, Mixture of Experts

Related terms

Ceiling, floor, round⌈x⌉ reads "ceiling of x" and rounds up. ⌊x⌋ reads "floor of x" and rounds down. The mixed ⌊x⌉ rounds to the nearest whole number.
𝓛 — the lossRead "L" or "the loss". The curly L is the single number that measures how wrong the model is. Training adjusts the weights to make it small.
FFN — the feed-forward networkRead "F F N of x". It is a small two-layer neural network applied to each token's vector on its own, with the same weights for every token.