Attention over time and channels

"You asked me to summarize the last ten minutes of vibration. I looked at all of it at once and decided which forty milliseconds actually mattered."

An Attentive AI Agent

Scaled dot-product attention: a content-addressed average

Attention takes a set of tokens, here a set of vectors extracted from the sensor stream, and rebuilds each one as a weighted blend of all the others. Every token is projected three ways by learned matrices into a query \(q\), a key \(k\), and a value \(v\). The query says what this token is looking for; the key says what each token offers; the value is the content it contributes if selected. Stacking the tokens of one window into matrices \(Q\), \(K\), and \(V\), the whole operation is:

$$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$

Read it inside out. The product \(QK^{\top}\) scores every query against every key by dot product, an \(n \times n\) grid of compatibilities for a window of \(n\) tokens. Dividing by \(\sqrt{d_k}\), the square root of the key dimension, keeps those scores from growing with dimension and saturating the softmax into a near one-hot spike; this is the scaled in the name, and it is what keeps gradients stable. The softmax turns each row into a probability distribution, the attention weights, and multiplying by \(V\) forms, for each token, a convex combination of all values weighted by relevance. The result is content addressing: attention is a lookup table the model writes at inference time, retrieving each token not by position but by resemblance.

A single attention map forces every relationship in the window through one weighting scheme, which is too coarse: a wrist IMU window may need one pattern to track a slow postural drift and a different one to catch a sharp tap transient, and one softmax cannot represent both at once. Multi-head attention fixes this by splitting the model dimension \(d\) into \(h\) smaller subspaces of size \(d/h\), running an independent scaled dot-product attention in each with its own \(Q\), \(K\), \(V\) projections, then concatenating the \(h\) outputs and mixing them with one further learned projection \(W_O\). Each head sees the same tokens through a different learned lens, so one head can specialize on the slow trend while another locks onto transients, at little extra compute since every subspace is narrower than \(d\). The payoff shrinks once the sensor set and window are themselves small, since a handful of channels over a short window may hold only two or three genuinely distinct relationships worth separating.

Attention over time: which moments explain each other

Point the mechanism along the time axis and each token is one time step (or, after Section 15.2, one short patch of time). The attention weights then form a map from each moment to every other, showing which instants the model consults to represent the present one, a strict generalization of what the temporal models of the previous chapter did. Where a causal convolution averages a fixed neighborhood with fixed weights, temporal attention averages the whole window with weights that depend on the signal itself, so a heartbeat can attend to the previous beat to measure an interval, and a gait classifier can line up the current stride against the one before it even if the cadence drifted. Because the raw operation is permutation invariant, it has no built-in sense of order; the model only learns that step \(t{-}1\) precedes step \(t\) once you add the position information of Section 15.3. For streaming and forecasting you also apply a causal mask that sets the scores for future positions to \(-\infty\) before the softmax, so a token attends only to the past, preserving the real-time constraint that disqualified bidirectional recurrence in Chapter 14.

The listing below runs one head of self-attention over a short multichannel window and reads off the temporal attention map, the row of weights that says where the model looks.

import torch, torch.nn.functional as F

torch.manual_seed(0)
T, C, d = 50, 6, 32                       # 50 time steps, 6 sensor channels, model width d
x = torch.randn(T, C)                     # one window: time by channel

Wq, Wk, Wv = (torch.randn(C, d) for _ in range(3))
Q, K, V = x @ Wq, x @ Wk, x @ Wv          # project each time step to query/key/value

scores = (Q @ K.transpose(0, 1)) / d**0.5 # T x T compatibility between every pair of steps
weights = F.softmax(scores, dim=-1)       # each row is a distribution over source steps
context = weights @ V                      # T x d : each step rebuilt from relevant steps

print("attention-map shape:", tuple(weights.shape))          # (50, 50): time attends to time
print("step 25 looks most at step:", int(weights[25].argmax()))

As Listing 15.1.1 shows, the attention map is a first-class, inspectable object: it tells you, per output moment, which input moments drove the representation.

Attention over channels: which sensors explain each other

Now transpose the question. Treat each sensor channel as a token, its value across the window (or a learned embedding of it) as the vector to project, and let attention run over the channel axis. The weights now form a sensor-by-sensor map: for a nine-axis inertial unit plus a barometer and a magnetometer, the model learns which channels inform which, so the three accelerometer axes can pool into a motion estimate while the magnetometer is down-weighted during an electrical disturbance. This is a learned, input-dependent alternative to the hand-built cross-channel features of Chapter 8, and its practical virtue is that it does not care about channel order or, within limits, channel count: a model attending over channels can absorb a sensor dropping out by masking its key rather than crashing on a missing input. Whether an architecture should mix channels at all, or keep them strictly separate, is a genuine design fork with real accuracy and robustness consequences; that decision is the subject of Section 15.5, so here we only establish that channel attention is the tool that makes mixing learnable.

import torch, torch.nn as nn

mha = nn.MultiheadAttention(embed_dim=64, num_heads=8, batch_first=True)
x = torch.randn(16, 50, 64)                       # batch, time steps, model width
out, attn = mha(x, x, x, need_weights=True)       # self-attention over the time axis
# feed x.transpose(1, 2) instead to attend over the channel axis

Putting the two axes together

Real sensor transformers rarely attend over one axis alone; the dominant patterns are three. Flatten time and channel into one long sequence of tokens and let full attention couple everything, the most expressive and most expensive option, since the score matrix grows with the product of the two axis lengths. Factorize the two axes into alternating blocks, a temporal-attention layer followed by a channel-attention layer, so cost stays linear in each axis while information still crosses both over depth; this axis-factorized design is the workhorse for multichannel sensing and reappears in the spatial-temporal graph models of Chapter 54. Restrict to one axis on purpose, the channel-independent choice, when the channels are better kept separate. The right pattern depends on how strongly your sensors are physically coupled and on your compute budget, a tradeoff of accuracy, latency, and memory the rest of this chapter equips you to navigate.

What's Next

In Section 15.2, we confront the quadratic cost head-on by changing what a token is: instead of one token per sample, we chop the stream into short patches and embed each as a single token, cutting the sequence length by the patch size and giving each token a richer local view before attention ever runs.