"I remember everything, which is exactly the problem. The trick was learning what to forget on purpose."
A Gated AI Agent
Prerequisites
This section assumes you are comfortable with sampled signals and the notion of a fixed-rate stream from Chapter 3, and with the neural-network mechanics (backpropagation, activations, minibatch training) refreshed in Appendix B and applied to sensor windows in Chapter 13. It is useful to have met the state-carrying, one-step-at-a-time structure of a recursive filter in Chapter 9, because a recurrent network is that same idea with a learned, nonlinear update rule. No prior exposure to sequence models is required.
The Big Picture
A recurrent neural network processes a sensor stream the way a Kalman filter does: it keeps a compact summary of everything seen so far, a hidden state, and updates it one sample at a time. Unlike the Kalman filter, it does not know the physics; it learns the update rule from data. This is a natural fit for sensing, where signals arrive as an ongoing stream and the meaning of the current sample depends on what came before it. The plain recurrent cell captures this idea in three lines of algebra but is notoriously hard to train over long horizons because gradients vanish. The LSTM and GRU are the two engineering fixes that made recurrence practical, each adding gates that let the network decide what to remember, what to overwrite, and what to expose. This section builds all three from the recurrence up, shows exactly why the plain version forgets, and gives you the judgment to pick the right cell for a sensor task.
The recurrent idea: a state that carries the past forward
A feedforward window model looks at a fixed slice of the stream and treats every slice independently. That works, but it forces you to commit to a window length up front and it throws away the connection between one window and the next. A recurrent network takes the opposite stance. It reads the stream sample by sample and maintains a hidden state vector \(h_t\) that is meant to summarize the relevant history. At each step it combines the new input \(x_t\) with the previous state to produce the next state:
$$h_t = \tanh(W_x x_t + W_h h_{t-1} + b)$$
The same weight matrices \(W_x\) and \(W_h\) are reused at every time step, which is the temporal analogue of the weight sharing a convolution performs across space. That sharing is what lets one small network handle sequences of any length and is the reason an RNN has a fixed parameter count no matter how long the stream runs. The why is compression: \(h_t\) is a lossy memory the network is trained to fill with exactly what the task will need later, and nothing else. For a wearable classifying gait, \(h_t\) might come to encode the phase of the current stride; for a fault detector it might encode how long a vibration has been elevated. The recurrence is trained by backpropagation through time: you unroll the loop across the sequence and treat it as a very deep feedforward network that happens to share weights across its layers.
Key Insight
A recurrent network is a state-space model whose transition function is learned rather than derived. The Kalman filter of Chapter 9 writes down \(h_t = A h_{t-1} + B x_t\) from known dynamics and gives you calibrated uncertainty for free. An RNN replaces \(A\) and \(B\) with a nonlinear learned map, buying the ability to model dynamics you cannot write down, at the cost of the interpretability and the built-in uncertainty. Choosing between them is the classical-versus-learned tradeoff that runs through this whole book, viewed one sample at a time.
Why plain RNNs forget: vanishing and exploding gradients
The elegance of the shared recurrence hides a defect that kept RNNs marginal for years. When you backpropagate the loss at step \(T\) back to an input at step \(t\), the gradient is a product of many Jacobians, one per step in between, each carrying a factor that involves \(W_h\) and the derivative of \(\tanh\). Multiplying \(T - t\) such factors together makes the gradient behave like a number raised to a large power: if the repeated factor has magnitude below one, the gradient shrinks toward zero (it vanishes); if above one, it grows without bound (it explodes). The practical consequence is that a plain RNN cannot learn dependencies that span more than a few dozen steps, because the error signal linking a late output to an early cause has decayed to nothing before it arrives.
For sensing this is a real ceiling, not a corner case. A five-second event sampled at 100 Hz is 500 time steps; a slowly developing bearing fault or a sleep-stage transition spans thousands. The exploding side is easier to patch (you clip the gradient norm to a threshold, which the training recipes of Section 14.7 treat in detail), but vanishing is structural: no learning-rate trick recovers a signal that has already decayed to numerical noise. This is precisely the problem the gated cells were invented to solve.
The listing below makes the decay concrete. It feeds a long random sequence through a plain RNN cell and an LSTM cell, then measures how much the loss at the final step still depends on the very first input.
import torch, torch.nn as nn
torch.manual_seed(0)
T, dim = 200, 16 # a 200-step sequence, e.g. 2 s of 100 Hz sensor data
x = torch.randn(1, T, dim, requires_grad=True)
for name, cell in [("vanilla RNN", nn.RNN(dim, dim, batch_first=True)),
("LSTM", nn.LSTM(dim, dim, batch_first=True))]:
x.grad = None
out, _ = cell(x) # run the recurrence over all T steps
out[:, -1, :].sum().backward() # loss depends only on the LAST step's output
# how strongly does the final-step loss still 'feel' the very first input?
first_step_grad = x.grad[:, 0, :].abs().mean().item()
print(f"{name:12s} gradient reaching step 0: {first_step_grad:.2e}")As Listing 14.1.1 shows, the difference is not a matter of tuning. The plain recurrence attenuates the long-range gradient by construction, and the LSTM's architecture is what preserves it. The next two subsections explain how.
LSTM: gates and a protected cell state
The Long Short-Term Memory cell adds a second state vector, the cell state \(c_t\), that acts as a memory conveyor belt. Information flows along \(c_t\) with only gentle, mostly additive edits, so gradients travel back along it without the repeated multiplicative shrinkage that kills the plain RNN. Each of its three gates is a small sigmoid layer, squashed to the zero-to-one range of a fraction: the forget gate \(f_t\) asks how much of the old memory to keep, the input gate \(i_t\) asks how much new candidate information \(\tilde{c}_t\) to write, and the output gate \(o_t\) asks how much of the memory to reveal as the visible hidden state, each computed from the current input and the previous hidden state:
$$f_t = \sigma(W_f x_t + U_f h_{t-1} + b_f), \quad i_t = \sigma(W_i x_t + U_i h_{t-1} + b_i), \quad o_t = \sigma(W_o x_t + U_o h_{t-1} + b_o)$$
$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t, \qquad h_t = o_t \odot \tanh(c_t)$$
Here \(\odot\) is elementwise multiplication, and the sigmoid \(\sigma\) is a deliberate choice: unlike a hard threshold, it stays differentiable everywhere, so gradient descent can nudge a gate's opening by a small amount instead of only flipping it fully open or shut, and its output doubles as a literal fraction of memory kept, written, or shown. Because every gate is recomputed from \(x_t\) and \(h_{t-1}\) at each step rather than following a fixed decay rate, the cell's retention policy is context-dependent: it can hold a value rock-steady through a quiet stretch of signal and release it the instant the input changes, something a fixed-rate exponential moving average cannot do. The payoff shows up in the cell-state equation above: when \(f_t\) is near one and \(i_t\) near zero, the cell copies its memory forward almost unchanged, giving the near-perfect gradient path known as the "constant error carousel." The network learns, per dimension and per time step, when to hold a value steady across hundreds of steps and when to let it change. For a clinical monitor tracking whether a patient has been tachycardic, one cell dimension can latch that fact and carry it for minutes while other dimensions track the fast beat-to-beat signal.
Common Misconception
The name "forget gate" leads many readers to expect that a high value erases the memory. It is the opposite: \(f_t\) close to one means keep the old cell state, and \(f_t\) close to zero is what actually erases it; the gate is named for the decision it makes, not for what a high output does. This mislabeling has a real training consequence: if the forget gate's bias is initialized near zero, the network defaults to erasing its memory at every step before it has learned any reason not to, which is why most modern LSTM implementations initialize that bias to a positive value (often 1), biasing the network to remember by default until training teaches it when to forget.
In Practice: an LSTM watching a haul-truck engine
A mining fleet operator instrumented its haul trucks with vibration, oil-temperature, and load sensors and wanted early warning of turbocharger degradation. A window CNN flagged failures only once they were loud, which was too late to schedule maintenance. The failure signature was a slow drift: elevated vibration that mattered only if it persisted and coincided with high load over many minutes, a dependency far longer than any practical convolution window. An LSTM fixed the problem because its cell state could accumulate the persistence evidence: one memory dimension effectively counted how long vibration had stayed above baseline under load, holding that count steady across the quiet stretches between rough patches of road. The forget gate learned to reset the count when the truck idled. This is the prognostics pattern developed fully in Chapter 36, and it is the kind of long-horizon, stateful memory that gating exists to provide; as always, the model was evaluated with truck-disjoint splits, the leakage-safe practice of Chapter 5, so that one vehicle's data leaking across train and test could not inflate the apparent lead time.
GRU: fewer gates, comparable power
The Gated Recurrent Unit is a streamlined cell that keeps the gating idea but drops the separate cell state. It merges the forget and input gates into a single update gate \(z_t\) that interpolates between the old state and a new candidate, and adds a reset gate \(r_t\) that controls how much past state feeds the candidate:
$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$
The convex combination is the crucial line: when \(z_t\) is near zero the state is copied forward unchanged, giving the GRU the same protected gradient path as the LSTM through a leaner mechanism. A GRU has three weight matrices where an LSTM has four, so it uses roughly 25 percent fewer parameters and runs a little faster for the same hidden size. On many sensor benchmarks the two are within noise of each other; the GRU tends to win when data or compute is tight, and the LSTM's extra capacity can pull ahead on the longest, most complex dependencies. For an edge deployment on a microcontroller (Chapter 61), the GRU's smaller footprint is often the deciding factor.
The Right Tool
Writing an LSTM by hand means implementing four gate projections, the cell-state update, the hidden-state readout, and correct minibatch handling of variable-length sequences, roughly forty lines that are easy to get subtly wrong (a swapped gate or a missing bias initialization can silently cripple training). Every framework ships all three cells as one line, fused and GPU-accelerated:
import torch.nn as nn
rnn = nn.RNN(input_size=6, hidden_size=64, num_layers=2, batch_first=True)
lstm = nn.LSTM(input_size=6, hidden_size=64, num_layers=2, batch_first=True)
gru = nn.GRU(input_size=6, hidden_size=64, num_layers=2, batch_first=True)Choosing a cell for a sensor stream
The decision is rarely "RNN, LSTM, or GRU" in isolation; it is a set of coupled choices. Reach for a plain RNN essentially never for real tasks, except as a teaching baseline or when the dependency is provably short and latency is critical. Between the gated cells, start with a GRU as the default (fewer parameters, faster, usually as accurate) and switch to an LSTM only if a validation comparison shows it winning on your longest dependencies. Set hidden size by the complexity of the state you expect to track, not by reflex: 32 to 128 units is often enough for sensor tasks, far smaller than language models. Stack two layers before you consider three, and make the network bidirectional only for offline analysis, because a bidirectional cell reads the whole sequence including the future and therefore cannot run in a live stream, a constraint that Section 14.5 makes central to streaming deployment. Finally, remember that recurrence is now one option among several: the temporal convolutions of Section 14.2, the transformers of Chapter 15, and the linear-time state-space models of Chapter 16 all compete for the same jobs, and the last of these is best read as the gated RNN's idea rebuilt for parallel hardware.
Research Frontier
The sequential dependency that makes an RNN hard to parallelize, each hidden state must wait for the one before it to finish, has become gating's main liability now that GPUs reward everything-at-once computation. A 2024 line of work ("minGRU" and "minLSTM") showed that if the gates are redefined to depend only on the current input \(x_t\) and not on \(h_{t-1}\), the recurrence collapses into a form a parallel scan can compute in \(O(\log T)\) rather than \(O(T)\) sequential steps, recovering most of the original accuracy while training on parallel hardware nearly as fast as the state-space models of Chapter 16. Whether this stripped-down gating keeps the full expressive power of the original LSTM on genuinely long, complex sensor dependencies is still an open empirical question.
Exercise
Build a synthetic "delayed recall" task: a sequence of random vectors where the label is a copy of the very first vector's sign, so the network must carry information from step 0 to step \(T\). (1) Train a plain nn.RNN, an nn.GRU, and an nn.LSTM at \(T = 20, 100, 500\) and plot final accuracy against \(T\). (2) At which \(T\) does the plain RNN collapse to chance while the gated cells still solve the task? (3) Add gradient-norm clipping to the RNN and report whether it helps the vanishing problem, the exploding problem, both, or neither, and explain why in terms of the Jacobian-product argument.
Self-Check
1. State the update equations for the plain RNN state and the LSTM cell state side by side, and point to the exact term in the LSTM equation that keeps long-range gradients from vanishing.
2. A GRU has fewer parameters than an LSTM of the same hidden size. Which gate combination is responsible for the reduction, and what capability, if any, do you give up?
3. Your task is a live wearable classifier that must emit a label every 20 ms as data arrives. Explain why a bidirectional LSTM is disqualified regardless of its offline accuracy.
What's Next
In Section 14.2, we set recurrence aside and build temporal convolutional networks, which attack the same long-sequence problem from the opposite direction: instead of carrying a state through time, they stack causal convolutions to see far into the past all at once, trading the RNN's sequential memory for parallelism and a fixed, analyzable receptive field.