A plain RNN forgets fast. Train one against an LSTM on a long-range task and watch the vanilla net's gradient die through time while the LSTM's cell-state path keeps memory alive.
rnnlstmbpttgradientsdeep-learningtrainingmemory
Live/Interactive · drag, toggle, run it
Deep learning, computed live
Recurrent Networks, End to End
A plain recurrent network forgets fast. Send a signal twenty steps down a sequence and it has usually faded to nothing by the time it matters, which is what made early RNNs so hard to train. LSTMs fixed it with three small gates: train one below, and watch its memory hold where a vanilla RNN's slips away.
What a hidden state actually is
A hidden state is not an abstraction, it is a vector of numbers the network keeps and rewrites at every step. This is a real 6-unit vanilla RNN, h_t = tanh(Wx x_t + Wh h_t-1 + b), with genuine weight matrices. Feed it a short sequence and read the hidden vector down each column of the heatmap: warm is positive, cool is negative. The recurrent matrix Wh decides whether that memory fades, holds, or saturates, and its largest eigenvalue magnitude is the dial.
Hidden state across time6 units, 8 steps
negative zero positive
largest |eigenvalue| of Wh
0.927near 1
|h| at final step
0.0908
|h| at first step
1.018
output head y
0.067
Memory roughly holds
The largest eigenvalue magnitude of Wh is 0.927, near one. The recurrent map neither shrinks nor blows up the carried state, so an early signal can persist across several steps without fading or saturating. This narrow band is the only place a plain RNN holds memory well, and a single shared matrix rarely stays in it across a whole task.
Controls
1.00
Scaling the recurrent matrix scales its eigenvalues. Drag below one to watch the hidden state die out, above one to watch it saturate.
Input sequence
The weight matrices behind the memory
Wh (recurrent, 6 x 6)
Wx (input, 6 x 1)
Wh maps the old hidden vector into the new one, then the input contribution Wx x_t is added and tanh squashes the result. Because the same Wh is applied at every step, repeated multiplication by it is what drives the carried memory. The largest eigenvalue magnitude (the spectral radius) is the asymptotic growth factor of that repeated map: below one it contracts, above one it expands. This is the honest matrix version of the single shared weight in the scalar vanishing-gradients story.