Practical training recipes

"My loss went to zero, my val curve looked gorgeous, and then a colleague pointed out that all my validation windows came from subjects I had already trained on."

A Chastened AI Agent

This section assumes the encoders and heads of Sections 14.1-14.4, the batching and masking of variable-length windows, and the normalization pipeline of Chapter 13. It leans on one non-negotiable discipline: the subject-disjoint, no-peeking-at-the-future evaluation of Chapter 5. Every recipe below is downstream of getting the split right; a tuning decision measured on a leaky validation set optimizes for a number that will not survive deployment.

The loss is the specification, not a default

The single most consequential training choice for sensor models is the loss, because sensor labels are almost always imbalanced. A fall detector sees minutes of walking for every fraction of a second of falling; a bearing runs healthy for months before it fails; an arrhythmia is a handful of beats in a day-long ECG. Train plain cross-entropy on that and the optimizer finds the cheapest solution: always predict "normal", collect 99.7% accuracy, and never fire. The fix is to make the rare class expensive to miss. Class-weighted cross-entropy multiplies each class's loss by a weight inversely proportional to its frequency. Focal loss goes further, down-weighting easy, confidently-correct samples by a factor \((1-p_t)^\gamma\) so the gradient concentrates on the hard minority, which usually beats static class weights for detection tasks. For heavily skewed segmentation, soft Dice or Tversky losses optimize overlap directly and are far less seduced by majority background than per-pixel cross-entropy.

Optimization for sequences: clip, warm up, decay

Recurrent networks propagate gradients through as many multiplications as there are timesteps, so a window of \(T=500\) samples is a 500-deep computation graph, and that depth makes the gradient norm volatile: one bad batch can produce a gradient a thousand times larger than the running average and blow the weights out in a single step. Gradient clipping is the cheap, load-bearing defense: clip the global gradient norm to a fixed ceiling (1.0 to 5.0 is a good starting range) before every optimizer step, and training that was diverging on batch 40 becomes stable. TCNs are less prone to this, since their gradients flow through a fixed convolutional depth, but clipping costs nothing and protects both. Note that clipping only caps a gradient that grew too large; it does nothing for one that decayed toward zero many steps back, the more common failure on long dependencies with a plain RNN, where the fix is architectural (gating, dilated convolutions) rather than a tighter clip threshold.

AdamW is the default optimizer here: it adapts per-parameter step sizes, which suits the heterogeneous gradient magnitudes of gated recurrent cells, and its decoupled weight decay is a cleaner regularizer than Adam's L2 coupling. Pair it with a schedule that warms up the learning rate over the first few hundred steps, then decays it with a cosine curve, because a large initial learning rate on a freshly initialized recurrent state produces exactly the gradient explosions clipping fights; easing in lets normalization statistics and gate biases settle first. The loop below also enables mixed precision via GradScaler: matrix multiplications run in float16 while master weights and the loss stay in float32, roughly halving activation memory and letting tensor cores run near double throughput. The risk is that small gradients underflow to zero in float16 before reaching the optimizer, silently starving the minority-class signal the loss above protects; the scaler compensates by scaling the loss up before backward and back down before clipping, the scale(...) and unscale_(...) calls below. Reach for it whenever GPU memory or throughput is the bottleneck; fall back to full precision only when debugging a numerically fragile loss.

import torch, math
from torch.optim import AdamW

def make_scheduler(opt, warmup, total):
    def lr_lambda(step):
        if step < warmup:                       # linear warmup
            return step / max(1, warmup)
        p = (step - warmup) / max(1, total - warmup)
        return 0.5 * (1 + math.cos(math.pi * p)) # cosine decay to 0
    return torch.optim.lr_scheduler.LambdaLR(opt, lr_lambda)

opt   = AdamW(model.parameters(), lr=3e-3, weight_decay=1e-2)
sched = make_scheduler(opt, warmup=300, total=10_000)
scaler = torch.cuda.amp.GradScaler()            # mixed precision

for x, y, mask in train_loader:
    opt.zero_grad()
    with torch.cuda.amp.autocast():
        loss = masked_loss(model(x), y, mask)   # ignore padded steps
    scaler.scale(loss).backward()
    scaler.unscale_(opt)
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    scaler.step(opt); scaler.update(); sched.step()

Regularization and augmentation that respect physics

Sensor models overfit fast because labeled sensor data is scarce, so regularization earns its keep. Weight decay and dropout are the baseline, but recurrent dropout must use the same mask across timesteps (variational dropout) rather than resample each step, or it corrupts the recurrence itself. The higher-leverage lever is data augmentation, and here sensor work diverges from vision: an augmentation is only legal if the label survives it. Gaussian noise, small time warps, magnitude scaling, and random cropping all preserve "this is walking". But flipping an accelerometer axis inverts gravity and turns walking into physical nonsense; time-reversing an ECG destroys the causal P-QRS-T ordering; rotating a three-axis IMU is valid only if all three channels rotate by the same matrix, since the axes are physically coupled. The measurement models of Chapter 2 tell you which transforms are label-preserving; borrow a vision augmentation blindly and you train on data that cannot occur.

Windowing, batching, and honest validation

Two mechanical choices decide whether your batches are even coherent. Window length must exceed the receptive field of Section 14.3 and must contain the temporal pattern the label describes; a two-second window cannot learn a gait cycle that takes three. Batching variable-length sequences requires padding to the longest member and a mask so the loss ignores the pad, exactly the masked loss the code above assumes; bucket similar-length sequences together to keep padding waste low. For sequences too long to fit in memory, truncated backpropagation through time splits the sequence into chunks, carries the recurrent state forward across chunk boundaries (detached from the graph), and backpropagates only within each chunk; set the truncation length to comfortably exceed the longest dependency the model needs, since gradients never flow past that boundary.

Of every choice in this chapter, the split decides the number that ships: get it right and a mediocre architecture still generalizes, get it wrong and a brilliant one just memorizes more convincingly. Random-window splits leak because consecutive windows from one subject or machine are nearly identical, so a validation window has a near-twin in training. Split by subject, by device, or by time period, never by window, and normalize using statistics computed on the training split alone. Early-stop on the leakage-safe validation metric and treat that curve as the truth. The formal protocols live in Chapter 65, and when labels are too scarce for any split to leave enough to train on, the self-supervised pretraining of Chapter 17 buys back label efficiency.

What's Next

In Chapter 15, we replace the recurrence and the fixed convolution with attention over time and channels, which lets a model relate any two timesteps directly, at a compute cost that forces its own set of training recipes: patchification, positional encoding for sensor identity, and the efficient-attention tricks that keep long sensor sequences affordable.