"I was handed a week of wrist data with the watch off for two nights, on the charger every afternoon, and a lost PPG lock during every run. My predecessor asked me to impute it first. I asked instead: why pretend the gaps are not there?"
A Pragmatic AI Agent
The big picture
Section 20.1 argued that raw multi-sensor streams deserve their own foundation models rather than being flattened into univariate time series. This section studies the most concrete realization of that idea to date: Google's Large Sensor Model (LSM) and its successor LSM-2, pretrained on tens of millions of hours of consumer wearable data. Their central lesson is not a clever architecture; it is a change of attitude toward missing data. Real wearable streams are incomplete by default: the device is charging, the sensor is off to save battery, the optical lock is broken by motion, the strap is loose. Classical pipelines impute those gaps and train as if the data were whole; LSM-2 instead makes missingness a first-class citizen of the pretraining objective through Adaptive and Inherited Masking (AIM), so the model learns directly from incomplete data and degrades gracefully as more of it goes missing at inference time.
This section assumes masked self-supervised pretraining from Chapter 17 and the Vision-Transformer patch-and-attend backbone of Chapter 15. It builds on the time-series foundation models of Chapter 19, generalized from one channel to a synchronized panel of heterogeneous sensors. The signals in play (PPG, accelerometry, electrodermal activity, skin temperature) are the wearable biosignals of Chapter 30.
What makes a model a "large sensor model"
What LSM does that a univariate time-series foundation model does not is treat a window of wearable data as a single two-dimensional object: sensor channels along one axis, time along the other. A day of minute-level data from a dozen derived channels becomes a grid, tokenized into patches exactly as a Vision Transformer tokenizes an image, then reconstructed under a masked-autoencoder objective. Why this framing matters is cross-channel structure: heart rate, motion, and skin temperature co-vary, and a model that attends across channels can infer a masked heart-rate patch from the accelerometer and temperature patches beside it. How the original LSM earned its scaling-law credibility: holding the architecture fixed, the team swept data volume, parameter count, and compute over roughly an order of magnitude each and plotted held-out reconstruction error on log-log axes, finding a straight line, a power law, in all three, the same signature that justified scaling language models. Why that matters practically: a predictable curve turns "should we train a bigger model" from a bet into an extrapolation, letting a lab forecast the return on a run it has not yet paid for. When to trust it: only within the swept range, up to roughly 40 million hours of data from over one hundred thousand consenting participants; nothing guarantees the same slope survives an architecture change or an order of magnitude beyond it. The payoff was strong zero-shot and few-shot performance on generative tasks (imputation, interpolation, extrapolation) without task-specific training.
Key insight: the panel is the token grid
Once you see a multi-sensor window as a channel-by-time image, the entire Vision-Transformer toolkit transfers: patch embeddings, positional encodings on both axes, masked reconstruction. The move a large sensor model makes over a time-series foundation model is not a new loss; it is a new substrate. Attention now flows across sensors as well as across time, so the pretrained prior is a joint prior over how physiological channels move together, not a bag of independent per-channel priors.
Missingness is the real problem, not an afterthought
The uncomfortable fact about wearable data is that it is almost never rectangular. Why gaps dominate: batteries force duty cycling, so a sensor may sample ten seconds a minute; charging removes the device for an hour a day; motion artifacts break the optical PPG lock during exactly the exercise you most want to measure; and different sensors run at different rates, manufacturing holes wherever a slow channel has no sample. A standard masked autoencoder assumes a complete input and adds synthetic masking as its learning signal; feed it real data and you face a dilemma: impute first (injecting a second model's errors into every downstream result) or drop incomplete windows (discarding most of your data and biasing the survivors toward users who wear the device perfectly). Neither is acceptable at foundation-model scale, and this is the same missing-modality challenge formalized for deep fusion in Chapter 50, here pushed to the point where the missingness is the majority of the signal.
Adaptive and Inherited Masking: learning from incomplete data
LSM-2's answer is to unify two kinds of masking under one mechanism. Inherited masking takes the missingness that already exists in the raw data (the device-off, sensor-off, artifact-corrupted cells) and marks those tokens as unobserved. Adaptive masking adds the artificial masking that self-supervision needs, but it adapts to what is already gone: it hides a target fraction of the genuinely observed cells rather than blindly masking a grid that is mostly holes. How the objective then works: the encoder sees only observed-and-unmasked tokens, and the decoder is asked to reconstruct only the adaptively masked targets, which are cells known to hold real values. The inherited-missing cells are never used as prediction targets, because there is no ground truth to score against, and they are never fed to the encoder as if they were real. Formally, with observation mask \(m_t \in \{0,1\}\) (one where a value truly exists) and adaptive mask \(a_t \in \{0,1\}\) (one where a real value is hidden for prediction), the loss averages only over cells that are both real and chosen:
$$\mathcal{L}_{\text{AIM}} = \frac{\sum_{t} \, m_t \, a_t \, \big(\hat{x}_t - x_t\big)^2}{\sum_{t} \, m_t \, a_t}.$$Misconception: the mask ratio is not a dial for missingness
It is tempting to read the mask ratio in the code below (0.75, familiar from vision MAE) as a dial you turn up to make the model tolerate more absence. It does not work that way: the adaptive-mask ratio applies only to cells that are genuinely observed (\(m_t = 1\)); raising it asks the model to reconstruct more of what it already has, and does nothing to the inherited-missing cells, which stay excluded from training regardless of the ratio chosen. Confusing the two masks is the most common place newcomers go wrong: adaptive masking is a training knob, inherited masking is a fact about the data no hyperparameter can override.
Why this beats impute-then-train is threefold. AIM never trains on a guess: only real values are ever asked to predict real values, so no imputation bias can leak into the representation. It trains on the exact input distribution the model meets at deployment, incomplete windows, not an idealized complete one. And because inherited and adaptive masks share one code path, the same forward pass learns from a nearly complete window and from one that is mostly empty. Across roughly two dozen downstream wearable tasks, LSM-2 reports not just higher accuracy than the first LSM but, more tellingly, far gentler degradation as inference-time missingness rises: drop half the sensors at test time and an AIM-trained model loses a fraction of what an impute-first pipeline loses.
Research frontier: where LSM and LSM-2 sit
Google's LSM ("Scaling Wearable Foundation Models", Narayanswamy et al., 2024) and LSM-2 (Adaptive and Inherited Masking, 2025) remain the largest publicly described sensor foundation models trained on raw consumer wearable streams, as opposed to the univariate corpora behind Chapter 19 models such as MOMENT or Chronos. Google has already extended the line with SensorLM (2025), pairing an LSM-style masked-grid encoder with a language model for text-grounded sensor reasoning, a preview of the multimodal sensor-language systems of Chapter 21. The open question: whether native-missingness pretraining, contrastive objectives, and generative masked modeling can combine into one backbone that serves both robust classification and faithful signal reconstruction. Watch this space alongside the Apple-style biosignal models of Section 20.3.
The bookkeeping below is what AIM actually requires in code: build the two masks, keep only the tokens that are both real and unmasked as targets, and never let a missing cell contaminate the loss. The function returns the target mask you multiply into the reconstruction error above.
import numpy as np
def aim_masks(observed, mask_ratio=0.75, rng=np.random.default_rng(0)):
"""Adaptive and Inherited Masking for a channel-by-time sensor grid.
observed: bool array (C, T), True where a real value exists (inherited).
Returns (encoder_input_mask, target_mask):
encoder sees observed AND not-adaptively-masked cells;
loss is scored only on target cells (observed AND adaptively masked)."""
C, T = observed.shape
adaptive = np.zeros((C, T), dtype=bool)
real_idx = np.argwhere(observed) # only mask real cells
n_target = int(round(mask_ratio * len(real_idx)))
chosen = rng.choice(len(real_idx), size=n_target, replace=False)
for c, t in real_idx[chosen]:
adaptive[c, t] = True
target_mask = observed & adaptive # real values to predict
encoder_input_mask = observed & ~adaptive # real values the model sees
return encoder_input_mask, target_mask
obs = np.random.default_rng(1).random((12, 1440)) > 0.4 # ~60% present
enc_mask, tgt_mask = aim_masks(obs, mask_ratio=0.75)
print(f"present cells: {obs.sum()}, encoder sees: {enc_mask.sum()}, "
f"targets: {tgt_mask.sum()}, missing never scored: {(~obs).sum()}")~observed) is excluded from both the encoder input and the loss targets; adaptive masking hides a fraction of the genuinely observed cells for the model to reconstruct. The returned target_mask is exactly the \(m_t a_t\) term in the loss.Library shortcut: masked-autoencoder plumbing
Hand-rolling the full pipeline (patch embedding, dual-axis positional encoding, encoder over visible tokens, lightweight decoder, per-cell loss gating) is roughly 250 to 400 lines. A masked-autoencoder scaffold from a library such as timm or the reference MAE implementation supplies the patchify, mask-token, and gather/scatter machinery, cutting the model definition to about 40 lines. You still write the AIM masking function yourself (about 15 lines): the inherited-versus-adaptive distinction is the sensor-specific part no vision library ships, precisely because images do not arrive with holes.
Practical example: a stress-detection wearable that keeps working with the strap loose
A digital-health team ships a smartwatch feature that flags elevated physiological stress from PPG, electrodermal activity, and skin temperature. In the lab every channel is clean; in the wild the electrodermal contact fails whenever the strap loosens, and PPG drops during typing bursts. Their first system imputed the missing electrodermal channel with a per-user mean and fed the completed grid to a classifier: accuracy looked fine on held-out lab data and collapsed in the field, because the imputed channel was confidently wrong exactly when stress spiked. Rebuilt on an LSM-2-style backbone fine-tuned with AIM, the model treats the loose-strap window as genuinely partial: it reads whatever real channels remain and abstains from inventing the missing one. Field accuracy recovers most of the lab gap, and the calibration curve, in the sense of Chapter 18, stays honest because no fabricated feature drives false alarms. The leakage-safe split behind this evaluation follows Chapter 5: users, not windows, are partitioned across train and test.
Exercise: measure graceful degradation
Take a pretrained masked sensor encoder (or the scaffold above) and a labeled downstream task with complete windows. Simulate deployment missingness by randomly zeroing out whole channels for a fraction \(p \in \{0, 0.2, 0.4, 0.6\}\) of test windows, using the observation mask rather than imputing. Plot downstream accuracy against \(p\) for two conditions: (a) the model fine-tuned with AIM-style masking, and (b) the same model with a mean-imputation front end. Report the slope of each curve. Which one degrades more gently, and does the gap widen as \(p\) grows? Explain the result in terms of where each pipeline injects error.
Self-check
- Distinguish inherited masking from adaptive masking in one sentence each. Which one supplies the self-supervised learning signal, and which one encodes real-world data absence?
- Why are inherited-missing cells excluded from the reconstruction loss rather than assigned a target of zero? What failure would a zero target cause?
- A colleague proposes imputing all gaps with a strong per-user model, then training a standard masked autoencoder on the completed grids. Name two concrete ways this can inflate offline metrics while hurting field performance.
What's Next
In Section 20.3, we turn to the other pole of wearable foundation models: Apple-style biosignal encoders that pretrain PPG, ECG, and accelerometer streams with self-supervision at population scale, and we compare their design choices (per-modality tokenization, participant-level contrastive objectives, on-device deployment constraints) against the unified masked-grid approach LSM and LSM-2 take here.