Continual learning for changing behavior

"They taught me to recognize running in March. In September the same person runs differently, and I have quietly forgotten how they walked. Nobody told me I was allowed to remember both."

A Forgetful AI Agent

Why behavior drifts and why retraining forgets

Two very different things get lumped together as "drift," and treating them the same causes trouble. In virtual drift (also called covariate shift) the incoming features \(P(x)\) change but the mapping from motion to activity, \(P(y \mid x)\), stays fixed: the same walking is now sampled from a differently worn watch. In real drift (concept drift) the relationship itself moves: the acceleration pattern that meant "jogging" for a healthy user comes to mean "brisk recovery walk" for the same user post-surgery, so the correct label for a near-identical signal has changed. The distinction matters because virtual drift can often be fixed by re-normalizing or re-calibrating the input, while real drift requires the model to learn a new decision, and only the second is a reason to change what the classifier believes.

The tempting response, fine-tune the network on whatever fresh labelled data arrives, runs straight into catastrophic forgetting. Gradient descent on a new task pulls every weight toward minimizing the new loss, with no term protecting the old one, so the shared representation is repurposed and accuracy on earlier activities collapses. This is one face of the stability-plasticity dilemma: a network plastic enough to absorb new behavior is, by the same token, plastic enough to destroy old behavior, and a network stable enough to never forget cannot learn anything new. Every method below is a different bargain struck at this exact tension.

Three families that fight forgetting

Continual-learning methods for activity recognition fall into three families, distinguished by what they protect. Replay (rehearsal) keeps a small memory of past examples and mixes them into every update, so the gradient always sees old activities alongside new ones. It is the simplest idea and, on inertial activity data, usually the strongest: a few hundred stored windows per class is often enough to hold the line, because a rehearsal buffer approximates the old data distribution directly rather than trying to summarize it. The cost is storage and, for wearables and health data, the privacy weight of retaining raw user motion.

Regularization methods store no data. They instead add a penalty that discourages moving the weights that mattered for old tasks. Elastic Weight Consolidation (EWC) is the canonical example: after learning, it estimates each parameter's importance with the diagonal of the Fisher information matrix \(F_i\) and, while learning the next task, penalizes movement away from the old optimum \(\theta^{\ast}_i\),

Important weights are held nearly rigid while unimportant ones stay free to absorb the new activity. Regularization is memory-light and privacy-friendly, but its protection is approximate and it degrades as many tasks accumulate.

The third family, parameter isolation, sidesteps interference by giving new behavior new capacity: a small adapter or a set of masked sub-network weights per context, freezing the rest. It forgets least but grows with the number of contexts, which suits the finite, known activity set of a fitness tracker better than an open-ended stream.

The listing below implements the workhorse of the first family: a class-balanced reservoir replay buffer that maintains a bounded, uniform sample of the stream so that rare activities are not crowded out by common ones.

import numpy as np

class ReplayBuffer:
    """Per-class reservoir sampling: a bounded, uniform memory of past windows."""
    def __init__(self, per_class=200, n_features=64, seed=0):
        self.per_class = per_class
        self.rng = np.random.default_rng(seed)
        self.store, self.seen = {}, {}          # label -> (X, count-seen)

    def add(self, x, y):
        if y not in self.store:
            self.store[y] = np.zeros((self.per_class, x.size), np.float32)
            self.seen[y] = 0
        n = self.seen[y]
        if n < self.per_class:                  # buffer not yet full: append
            self.store[y][n] = x
        else:                                   # full: replace with prob k/n
            j = self.rng.integers(0, n + 1)
            if j < self.per_class:
                self.store[y][j] = x
        self.seen[y] = n + 1

    def sample(self, k):
        xs, ys = [], []
        for y, X in self.store.items():
            m = min(self.seen[y], self.per_class)
            for i in self.rng.integers(0, m, size=max(1, k // len(self.store))):
                xs.append(X[i]); ys.append(y)
        return np.stack(xs), np.array(ys)

buf = ReplayBuffer(per_class=200)
for x, y in [(np.random.randn(64), lbl) for lbl in np.random.randint(0, 5, 5000)]:
    buf.add(x.astype(np.float32), int(y))       # stream one window at a time
Xr, yr = buf.sample(128)                         # mix these into each new update
print("rehearsal batch:", Xr.shape, "classes:", np.bincount(yr))

As Listing 26.5 shows, the mechanism that beats forgetting is modest: bounded memory plus balanced sampling. The subtlety is entirely in the sampling policy, which is why hand-rolled buffers tend to quietly starve the classes you most wanted to protect.

Evaluating continual learning without fooling yourself

A single accuracy number cannot describe a model that changes over time, and reporting only the final one hides exactly the failure this section is about. Evaluate along the time axis with three quantities computed on held-out data from every stage the model has passed through. Average accuracy after learning stage \(T\) is the mean over all stages \(j \le T\) of accuracy on stage \(j\)'s test set. Backward transfer (forgetting) measures how much learning later stages hurt earlier ones: with \(a_{T,j}\) the accuracy on stage \(j\) after training through stage \(T\),

where a large negative value is catastrophic forgetting and a small positive value means later behavior actually reinforced earlier competence. Forward transfer asks the complementary question: does what the model already learned make it faster to pick up the next behavior? Measure it by comparing accuracy on stage \(T\)'s task after a few gradient steps from the continual model against the same steps from a randomly initialized one; a positive gap means earlier stages built reusable structure, shared gravity-axis alignment, gait periodicity, that the new activity can draw on. It matters because BWT alone can flatter a method that only avoids interference: a replay buffer can hold BWT near zero while contributing nothing to how fast stage three is learned. Check forward transfer whenever a new context should resemble the old ones, a new gait variant rather than an unrelated activity, since that is exactly when reusable structure should pay off. The right online protocol is prequential (test-then-train): predict and score each incoming window, then use it to update, so every example is a genuine test before it is ever a training point. The leakage rule from Chapter 65 bites with special force here: because consecutive windows are correlated in time, a random split leaks the near-future into training and makes a forgetful model look stable. Split by contiguous time and by subject, and never let a replay buffer's contents leak into the test set. Continual learning is the temporal sibling of the test-time adaptation and distribution-shift methods of Chapter 66; the difference is that the model keeps the update, so measuring what it lost matters as much as what it gained.

What's Next

In Section 26.6, we widen the lens from one changing user to a whole population. A model that learns continually still learns from whoever generates the most data, and behavior varies not only over time but across bodies, ages, and demographics. We take up fairness across users: how activity recognizers come to work well for some groups and poorly for others, how to measure that gap, and how to close it.