Vanishing and Exploding Gradients
Stack enough layers and the gradient either fades to nothing or explodes. Tune depth, activation, and weight scale, watch the per-layer gradient norm decay or blow up, then see it happen through time in an RNN.
Vanishing and Exploding Gradients
Backprop sends one gradient backward through a deep chain by multiplying, layer by layer, by a local factor: the activation derivative times the weight. Multiply many factors below one and the gradient collapses before it reaches the first layer; multiply many above one and it explodes instead. Tune the chain below and watch which way it tips.
Backprop starts at the output (right) and walks left toward the input. Step it to watch the running gradient pick up one factor of act' x weight per layer, the product the chain rule is assembling.
sigmoid: derivative peaks at 0.25 and saturates to 0 in the tails.
The gradient at the input layer is 2.41e-11 times the gradient at the output. Each layer multiplies by a factor near 0.1949, below one, so across 16 layers the signal decays to almost nothing. The early layers barely learn. Saturating activations make this worse because their derivative caps at 0.25, and a large input drives them into the flat tails where it falls toward zero.
The chain rule writes the gradient at an early layer as a product of every Jacobian between it and the loss. For this scalar chain that product is the running multiplication of act'(z) times the weight, layer after layer. If the typical factor is below one, the product decays like factor to the power of depth; if it is above one, it grows the same way. Both are exponential in depth, which is why depth and long time horizons are exactly where the problem bites. Sigmoid and tanh make the vanishing side worse: their derivatives cap at 0.25 and 1 and fall to zero once a unit saturates, so most factors start well below one before the weights are even counted.
The fixes all attack the factor. ReLU keeps the derivative at exactly one on its active branch, so it does not shrink the signal the way a saturating unit does. Xavier and He initialization set the weight variance so the typical factor lands near one at the start of training, which is the dynamical-isometry idea: keep the chained Jacobian close to an isometry. Batch and layer normalization hold activations in the responsive part of the nonlinearity so derivatives stay healthy. Residual connections add an identity path, so the gradient has a route that multiplies by one and reaches early layers intact. For recurrent nets the same repeated factor across time is what LSTMs and GRUs address: their gating gives the cell state a near-identity path through time, letting gradients survive across many steps instead of decaying as factor to the power of T.