Neuron — a weighted sum and an activation
The smallest unit of a neural network: a weighted sum of its inputs plus a bias, passed through a fixed nonlinear function.
A layer is many neurons reading the same input, which is one matrix multiplication followed by the activation function applied entry by entry (section 4.2).
Where the course uses it
Related terms
- Activation function — the nonlinearityThe fixed nonlinear function applied after a layer's weighted sums, such as ReLU, tanh, GELU or Swish.
- MLP — multi-layer perceptronA chain of layers, each a matrix multiplication followed by an activation function. The feed-forward network in every transformer block is an MLP with one hidden layer.