Segmentation and transition states

"I was ninety-eight percent sure the user was sitting and ninety-eight percent sure they were standing. The half second in between, when they were neither, is the only part anyone ever asks me about."

An Over-Confident AI Agent

Why the continuous stream resists cutting

The dominant approach is the sliding window: slice the stream into fixed-length frames of length \(W\) with hop \(H\), extract features from each, and classify each frame independently. It is simple, streams naturally, and dominates the literature. It also hides an unavoidable tension in the choice of \(W\). A window must be long enough to contain the periodic structure of the activity: a full gait cycle for walking is roughly one second, so a \(0.5\ \mathrm{s}\) window literally cannot see a stride and will confuse walking with fidgeting. But a long window is more likely to straddle a boundary and contain two activities at once, which no single label describes. Every window length trades one error for another: a short window sees the boundary but not the activity, a long window sees the activity but not the boundary, and no fixed \(W\) escapes both failures at once.

Formally, with hop \(H\) the fraction of windows that are "mixed" (straddle a true boundary) grows with \(W\): the expected number of boundary-straddling windows per segment is about \(W/H\), independent of the segment's own duration \(\bar{d}\), so short activities suffer most. A two-second bout of stair-climbing sampled with a three-second window may never produce a single pure window. Smaller \(H\) improves temporal resolution and yields more training examples, but also makes adjacent windows highly correlated, which quietly inflates any accuracy number computed by splitting windows randomly rather than by subject or by time.

Two families of segmentation

Segmentation strategies split into two families. Explicit (bottom-up) segmentation first finds boundaries, then classifies the resulting variable-length segments. Boundaries come from change-point detection on the signal statistics: a sit-to-stand event is a sharp change in the mean and variance of vertical acceleration, exactly what the CUSUM and sliding-window detectors of Chapter 12 are built to catch. The appeal is that each classified segment is internally homogeneous, so the classifier never sees a mixed window. The risk is cascading error: a missed boundary merges two activities forever, and a detector run on raw acceleration will fire on every footstep unless smoothed or thresholded carefully.

The listing below applies an off-the-shelf change-point search to a synthetic accelerometer-magnitude trace with three regimes (still, walking, still) and recovers the two boundaries.

import numpy as np
import ruptures as rpt

fs = 50.0
t = np.arange(0, 12, 1 / fs)
sig = np.full(t.size, 9.81)                     # accel magnitude, resting
walk = (t > 4) & (t < 8)                        # a walking bout
sig[walk] += 2.5 * np.sin(2 * np.pi * 1.8 * t[walk])   # ~1.8 Hz gait
sig += np.random.normal(0, 0.15, t.size)

# Pelt search on a shift in mean+variance; penalty controls sensitivity.
algo = rpt.Pelt(model="rbf", min_size=int(1.0 * fs)).fit(sig)
bkps = algo.predict(pen=12.0)                   # indices of change points
print("boundaries at t =", [round(b / fs, 2) for b in bkps[:-1]], "s")

As Listing 26.3 shows, boundary-finding can be a few lines rather than a hand-rolled CUSUM. The second family, implicit (dense) segmentation, skips explicit boundaries entirely: a sequence model emits a label at every timestep, and boundaries appear wherever the predicted label changes. Recurrent and temporal-convolutional networks from Chapter 14 do this naturally, learning transition dynamics from data instead of a hand-tuned detector. This is now the default for high-quality systems because it makes the boundary a learned, context-aware decision rather than a threshold on one statistic. Its cost is data hunger: it needs densely labelled streams, far more expensive to annotate than isolated clips.

Evaluating a segmentation without fooling yourself

Segmentation quality cannot be judged by frame accuracy alone, and here the running leakage theme from Chapter 65 bites twice. First, frame-level accuracy is dominated by long, easy, stationary spans and is nearly blind to the short transitions you most care about; a model can score 94% frame accuracy while missing half of all sit-to-stand events. Report event-level metrics too: count a predicted segment as a true positive only if it overlaps the correct ground-truth segment above some threshold, and break failures down by the classic segmentation error taxonomy rather than one F1 number. An insertion is a predicted segment with no matching ground truth, a false transition conjured from noise; a deletion is a true segment the model never found, such as a missed sit-to-stand; a merge fuses two adjacent true segments into one because the model smoothed through a short bout; and fragmentation splits one true segment into several because a noisy signal made the model flicker mid-activity. Insertions and fragmentation mean the detector is too twitchy (raise the penalty, add hysteresis); deletions and merges mean it is too sluggish (lower the penalty, shorten the window); the aggregate F1 alone hides which knob to turn. Second, because overlapping windows correlate strongly in time, a random train/test split leaks near-identical neighbours across the split and inflates every number: always split by subject and by contiguous time block, never by window.

A subtler point concerns the boundaries themselves: human annotators disagree about exactly where a transition starts by a few hundred milliseconds, so demanding frame-exact agreement penalizes a model for beating the noise floor of its own labels. Score boundaries with a tolerance window instead (a predicted boundary within, say, \(\pm 0.5\ \mathrm{s}\) of a labelled one counts as correct), matching the intrinsic imprecision of the ground truth rather than pretending it is exact.

What's Next

In Section 26.4, we turn from where to cut the stream to whose stream it is. Segmentation and classification both degrade when a model trained on one population meets a new user whose gait, pace, and transition style differ, so we take up user personalization and adaptation: how to bend a general model toward the individual wearing it without collecting a fresh labelled dataset for every person.