An LSTM remembers across gaps where a plain RNN forgets. Open one cell, run it on a sequence, and watch the forget, input, and output gates carry memory along the cell state's near-identity highway.
lstmrnngatesgradientsdeep-learningmemory
Live/Interactive · drag, toggle, run it
Deep learning, computed live
Inside an LSTM Cell
A plain recurrent net rewrites its whole memory at every step, so a gradient flowing back through time gets multiplied by a Jacobian over and over until it vanishes. An LSTM keeps a separate cell state and learns three gates that decide what to erase, what to write, and what to read. Step one cell through a short sequence below and watch the cell state carry information forward while the gradient still has a path home through time.
Task preset
A high forget bias pins the gate open near 1, the input gate stays shut after the first step, and the cell state carries that opening value almost unchanged to the end.
The cell at step 11 hidden unit, 1 input per step
The thick line is the cell state, the memory highway. The forget gate (x) pinches it; i times g is added in. The line thickens as the forget gate opens toward 1.
fforget0.99
iinput0.82
gcandidate0.85
ooutput0.90
input x(t)
0.90
kept f * c(t-1)
0.00
written i * g
0.70
cell state c(t)
0.70= kept + written
output h(t)
0.54o * tanh(c)
step 1 / 8
Cell state and output across the sequence
cell state c(t) hidden output h(t)
Input sequence
One scalar per step. Edit any value, or add and remove steps to make the sequence longer or shorter.
Gate weights
5.0
Raising the forget bias pushes the forget gate toward 1, so the cell holds its memory across steps. This is why LSTMs are often initialized with a positive forget bias.
-3.0
Lowering the input bias keeps the input gate near 0, so the cell stops overwriting itself and protects what it already holds.
The remaining weights come from the chosen preset. Each gate computes its pre-activation as a weight on the input, a weight on the previous hidden value, and a bias, then a sigmoid (f, i, o) or tanh (g).
The four gates, in one place
f · forgetf = sigmoid(W_f x + U_f h + b_f)
how much old memory to keep.
i · inputi = sigmoid(W_i x + U_i h + b_i)
how much new candidate to write.
g · candidateg = tanh(W_g x + U_g h + b_g)
the value proposed for writing.
o · outputo = sigmoid(W_o x + U_o h + b_o)
how much of the cell to expose.
the cell update, protected memory
c(t) = f · c(t-1) + i · g h(t) = o · tanh(c(t))
The cell state is updated by addition, not by a full rewrite. The forget gate scales the old memory, the input gate scales the new candidate, and the two are summed. When forget is near 1 and input is near 0, the cell state passes through almost unchanged.