RNNs & Vanishing Gradients
A standard neural network treats every input independently. But language, audio, and time- series data are sequential — order matters, and earlier inputs influence later ones. Recurrent Neural Networks (RNNs) address this by maintaining a hidden state that is passed from one time step to the next: ht = tanh(Wh·ht-1 + Wx·xt + b).
This hidden state acts as a kind of "memory." But it comes with a fatal flaw: the Vanishing Gradient Problem. During backpropagation through time (BPTT), the gradient is multiplied by the same weight matrix at every step. If the recurrent weight has magnitude less than 1, the gradient shrinks exponentially — the network forgets. If greater than 1, it explodes, causing instability.
The solution came with Long Short-Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997). LSTMs introduce a cell state — a direct "highway" for gradient flow — controlled by three gates: the forget gate (what to discard), the input gate (what new information to store), and the output gate (what to expose). GRUs (Gated Recurrent Units) are a simpler, often equally effective variant. Both are now largely superseded by Transformers for most tasks.