Activation Functions, Curve by Curve
Sigmoid and tanh flatten at the edges, and their gradient dies there. Drag a point across the curves to compare slopes, and watch the ReLU family keep the gradient alive where the others go silent.
Activation Functions, Curve by Curve
An activation is the nonlinearity a neuron applies after its weighted sum. Without it a stack of layers collapses into one linear map, so the network could never bend a decision boundary. Its derivative is what backprop multiplies as the gradient flows back, so the shape of that derivative quietly decides how well a deep net learns. Overlay a few below and watch where each one keeps its slope alive and where it flatlines.
A derivative near zero is a saturation region: the unit barely responds, so almost no gradient passes through it. For ReLU every x below 0 has derivative exactly 0, the dead zone where a unit stuck on the negative side stops updating. Leaky ReLU and ELU keep a nonzero slope there, so the gradient survives.
Backprop pushes the gradient back through every layer by multiplying by each unit's local derivative. Stack the chosen activation 8 layers deep at this operating point and the factor that reaches the first layer is the product of 8 copies of |f′(x)|. When that factor is below 1 it shrinks geometrically, the vanishing-gradient problem; the ReLU family holds it at 1 on the active half.
Sigmoid caps its derivative at 0.25, so even at its steepest point a stack multiplies by at most 0.25 each layer and the gradient collapses fast. Tanh peaks at 1 only at the origin and falls off on either side. ReLU on its active half keeps each factor at exactly 1, so the product stays at 1 no matter how deep the stack.
| Function | Output range | Derivative range | Zero-centered | Saturation | Dead units |
|---|---|---|---|---|---|
| Sigmoid | (0, 1) | (0, 0.25] | no | Both tails flatten; derivative to 0 | No hard dead zone, but tails barely learn |
| Tanh | (-1, 1) | (0, 1] | yes | Both tails flatten; derivative to 0 | No hard dead zone, but tails barely learn |
| ReLU | [0, infinity) | {0, 1} | no | Negative half is exactly flat | Yes: a unit pushed below 0 gets zero gradient |
| Leaky ReLU | (-infinity, infinity) | {0.1, 1} | no | Never flat; negative slope stays nonzero | No: negative side passes 0.1 of the gradient |
| ELU | (-1, infinity) | (0, 1] | no | Saturates softly to -alpha on the left | No: negative side keeps a smooth nonzero slope |
| GELU | approx (-0.17, infinity) | approx (-0.13, 1.08) | no | Left tail to 0; right tail to 1 | No: small negative values still pass gradient |
| SiLU / Swish | approx (-0.28, infinity) | approx (-0.10, 1.10) | no | Left tail to 0; right tail to 1 | No: smooth nonzero gradient near and below 0 |
GELU uses the exact erf form, f(x) = 0.5 x (1 + erf(x / sqrt 2)); SiLU is x times sigmoid(beta x). Their derivative ranges dip slightly below zero because each has a shallow dip just left of the origin, which is why their stated ranges are approximate.
Zero-centered activations like tanh let a layer push its output both above and below zero, so the next layer's gradients are not all forced to share a sign; sigmoid, stuck in (0, 1), biases every downstream update the same way and slows learning. Non-saturating activations matter even more: once a sigmoid or tanh unit lands in its flat tail, its derivative is near zero and it stops contributing to the gradient, so depth multiplies many small factors into nothing.
ReLU avoids saturation on its active half, which is why it unlocked very deep networks, but it pays for that with a hard dead zone on the left. Leaky ReLU, ELU, GELU, and SiLU each keep a nonzero slope below zero so a unit can recover, trading a little of ReLU's simplicity for gradients that never fully die. Drag the input into the tails above and the readouts make the tradeoff concrete.