Sequence-to-label and sequence-to-sequence

"I spent a week improving the encoder, then discovered I had been averaging over the one second where the fall actually happened."

A Belatedly Wiser AI Agent

This section assumes you can build the encoders from Section 14.1 and Section 14.2 and understand the receptive-field arithmetic of Section 14.3, because the receptive field determines how much context each per-timestep output can see. Prerequisites also include cross-entropy and masked losses from the deep learning refresher (Appendix B) and the subject-disjoint, no-peeking-at-the-future evaluation discipline of Chapter 5. The output topology and the evaluation protocol are joined at the hip.

Three output topologies, one encoder

Given an encoder that maps an input sequence \(x_{1:T}\) to hidden states \(h_{1:T}\), there are three ways the output can relate to the input. Sequence-to-label emits a single prediction \(y\) for the whole window: is this ten-second accelerometer clip walking or falling, what is the remaining useful life of this bearing. Aligned sequence-to-sequence emits one label per input step, \(y_{1:T}\), where output \(t\) corresponds to input \(t\): is the person walking at this sample, is this ECG sample inside a QRS complex. Unaligned sequence-to-sequence emits an output \(y_{1:U}\) whose length \(U\) differs from \(T\) and whose steps do not line up with input steps: transcribe a stroke of handwriting into characters, forecast the next \(U\) samples from the past \(T\). The encoder can be identical in all three cases: only the head and the loss change.

Sequence-to-label: pooling a window into one answer

To collapse \(h_{1:T}\) into one vector you need a pooling operator, and the choice matters. The classic RNN move is last-state pooling: take \(h_T\) and discard the rest, on the theory that a recurrent state has already summarized the past. It is cheap and it is what most tutorials show, but it is fragile for sensors, leaning hardest on the most recent samples, so a fall that happens mid-window can be washed out by two seconds of lying still afterward. Mean pooling averages \(h_{1:T}\), treating every timestep as equally informative: robust, but it dilutes short, sharp events. Max pooling takes the per-feature maximum and suits "did this pattern appear anywhere" detection, such as spotting a single arrhythmic beat. Attention pooling learns a weight \(a_t\) per timestep and forms \(\sum_t a_t h_t\), letting the model concentrate on the informative instants in a single differentiable head; it is usually the best default, and the weights \(a_t\) double as a free saliency map over time that the interpretability tooling of Chapter 67 can read directly.

A subtlety with variable-length windows: your batch is padded to the longest sequence, so pooling must mask the padding. Averaging over padded zeros silently biases the estimate toward whichever class produces short sequences. The code below shows masked mean pooling and attention pooling side by side.

import torch, torch.nn.functional as F

def masked_mean(h, mask):                      # h: (B,T,D)  mask: (B,T) 1=real 0=pad
    m = mask.unsqueeze(-1).float()
    return (h * m).sum(1) / m.sum(1).clamp(min=1)

def attention_pool(h, mask, score):            # score: Linear(D, 1)
    logits = score(h).squeeze(-1)              # (B,T) one relevance logit per step
    logits = logits.masked_fill(mask == 0, float('-inf'))
    a = F.softmax(logits, dim=1).unsqueeze(-1) # (B,T,1) weights sum to 1 over real steps
    return (a * h).sum(1), a.squeeze(-1)       # pooled vector + saliency over time

B, T, D = 4, 50, 16
h = torch.randn(B, T, D)
mask = torch.ones(B, T); mask[0, 30:] = 0      # first clip is only 30 steps long
pooled_mean = masked_mean(h, mask)
pooled_attn, saliency = attention_pool(h, mask, torch.nn.Linear(D, 1))
print(pooled_mean.shape, pooled_attn.shape, saliency.shape)

The pooled vector then feeds a small classifier or regressor: a linear layer plus cross-entropy for classification, a linear layer plus a regression loss for a scalar target such as remaining useful life. Whatever the pooling, apply the exact same masking at train and inference time, or the model will see a distribution at deployment it never trained on.

Aligned sequence-to-sequence: one label per timestep

When every timestep carries its own label, drop the pooling and apply the head per step: a shared linear-plus-softmax on each \(h_t\), trained with cross-entropy averaged over all valid (unpadded) timesteps. This is the natural shape for temporal segmentation: labeling each ECG sample as P-wave, QRS, or T-wave; marking each accelerometer sample for the boundary-precise human activity recognition of Chapter 26; flagging each sample as normal or anomalous, the dense cousin of the change detection in Chapter 12.

The one rule that governs quality here is causality of the receptive field. A per-step output at time \(t\) is only as good as the context it can see. A bidirectional LSTM or a non-causal TCN lets output \(t\) peek at future inputs, which sharpens offline segmentation but is illegal for real-time streaming, where output \(t\) must be emitted before input \(t+1\) exists. The receptive-field budget from Section 14.3 tells you exactly how many past samples each aligned output integrates; if a transition takes longer than that budget to become apparent, the head cannot possibly label its onset correctly, and the fix is a deeper dilation stack, not a fancier loss.

Unaligned sequence-to-sequence: when input and output lengths differ

Some tasks produce an output whose length is not the input's and whose steps do not line up one-to-one: transcribing an inertial pen trajectory into characters, decoding surface-EMG into intended keystrokes for the neuromotor interfaces of Chapter 27, or forecasting the next \(U\) samples of a signal. Two families handle this. Encoder-decoder models compress the input into a context (a final state or a set of encoder states), then generate the output autoregressively: at each step a decoder consumes that context plus its own previous output and predicts the next token, optionally attending back over the encoder states. Paying for this machinery over a simpler alignment-free loss buys conditioning on the outputs already produced, which matters whenever output tokens depend on each other (the next character in a word is constrained by the characters already written); the price is sequential, error-compounding generation. This topology grows into the attention-based sequence models of Chapter 15. Alignment-free losses, chiefly Connectionist Temporal Classification (CTC), let a per-step encoder emit a longer frame-level label stream (with a blank symbol) and marginalize over all alignments that collapse to the target, so you get a sequence output without ever labeling which input frame produced which output token, exactly the label you cannot afford to collect for handwriting or myoelectric typing.

Forecasting deserves a note: it is the sensor world's most common unaligned task, and it hides a leakage trap. Predicting \(x_{T+1:T+U}\) from \(x_{1:T}\) must be trained and evaluated with a strictly causal split: the target window is the future, and any normalization statistic, feature, or neighbor drawn from after \(T\) leaks the answer. The horizon \(U\) also interacts with uncertainty: a point forecast far out is nearly useless without a calibrated interval, the job of Chapter 18.

What's Next

In Section 14.5, we take the aligned per-step head into the hardest regime it faces: streaming and stateful inference, where the model must emit output \(t\) from a running state before sample \(t+1\) arrives, carry that state across window boundaries without discontinuities, and match its offline-trained behavior exactly on an endless live signal.