Device, user, site, session, and machine splits

"The test set exists to answer one question: what happens when something new walks in the door. If the door already knows the visitor, you asked a different question and reported the wrong answer."

A Disciplined AI Agent

You cut the streams into windows back in Section 5.2 and learned every way they leak in Section 5.3; now picture shipping a model that scored 97% in the lab, watching it drop to a coin flip the instant a new patient is wheeled in, and discovering that the only thing standing between triumph and embarrassment was where you drew the line between train and test. This section turns that diagnosis into a construction procedure: we take a windowed, labeled dataset carrying entity identifiers and produce train, validation, and test partitions whose accuracy estimate matches the deployment question you actually care about.

The grouping unit is a modeling decision, not a default

A model that has quietly learned to recognize the wearer instead of the motion can post a near-perfect lab score and then collapse the moment a stranger straps on the sensor, and the whole difference traces to a single decision about how you group the data. What. A grouping unit is the entity across whose boundary your model must generalize at deployment. Every window belongs to exactly one value of that unit: this wearer, that machine, this recording session. A group-disjoint split then guarantees that all windows sharing a group value land entirely in train or entirely in test, never straddling the two. Why. Sensor data is riddled with nuisance signals that are constant within an entity and irrelevant to the label: a specific accelerometer's bias, one person's gait idiosyncrasy, a particular lathe's bearing hum, a room's acoustic reverberation. A random split scatters an entity's windows across train and test, so the model can lower its loss by recognizing the entity rather than the phenomenon. That shortcut evaporates the instant a genuinely new entity appears. In short: pick the entity your model must generalize across, keep every one of its windows on a single side of the split, and your accuracy finally means what it claims. Figure 5.4.2 illustrates random window split versus group-disjoint split (how leakage arises).

How. Pick the unit by asking a single question: at deployment, what will be new? The answer names your holdout axis. The common units, roughly ordered from finest to coarsest, are:

• Session (recording). One continuous capture from one entity. Holding out sessions defeats the crudest leakage (adjacent overlapping windows) but still lets the same person or device appear in train and test.
• Device. One physical sensor unit. Device-disjoint splits test whether the model survives manufacturing tolerance, mounting variation, and per-unit calibration drift (the subject of Section 5.5).
• User / subject. One person wearing or operating the sensor. Subject-disjoint splitting is mandatory for any wearable, biosignal, or activity recognition model, because human physiology and behavior are the dominant nuisance variables.
• Machine. One physical asset in industrial monitoring: a pump, a turbine, a computer numerical control (CNC) spindle. Fault models must generalize to machines never seen during training.
• Site. A whole location: a hospital, a factory floor, a building. Site-disjoint splits probe environmental and installation shift, the coarsest and usually hardest generalization gap.

Nested entities and the strictest-boundary rule

Matching one split to one sentence is clean when a single entity varies, but real deployments rarely grant that simplicity. Real datasets carry several nested identifiers at once. A wearable study has subject ⊃ device (a loaner watch) ⊃ session (each day's wear) ⊃ window. These nest: every session belongs to one device, every device to one subject. The rule: split at the coarsest boundary your deployment claim requires, and the finer boundaries take care of themselves automatically, because holding out a subject also holds out all of that subject's devices and sessions. Figure 5.4.1 shows why: once a whole subject is sent to one side of the split, every device, session, and window nested inside it follows along, so no finer identifier can straddle the boundary.

The danger is a partial identifier that silently couples the two sides. If one subject wore two devices and you split by device, that subject's physiology now appears in both partitions even though no device does; you have a subject leak hiding inside a clean-looking device split. Formally, a split into train set \(\mathcal{T}\) and evaluation set \(\mathcal{E}\) is group-safe for unit \(g\) when their group sets are disjoint:

When several units matter at once (say subject and site), the safe construction holds out on the composite key, or splits on the coarsest unit that dominates the others. Verify the intersection is empty for every unit you claim to generalize across, not just the one you sorted on.

Constructing grouped splits in practice

Having settled which boundary to draw and why, the remaining work is purely mechanical. The mechanics are short once the group vector exists. You never shuffle windows; you shuffle groups, then gather every window of the chosen groups. For cross-validation you want each fold's test groups disjoint from its training groups, which is exactly what grouped K-fold provides. The snippet below shows a manual subject-disjoint split and the grouped-CV equivalent, and prints the group overlap so you can prove disjointness rather than assume it.

Grouped K-fold is ordinary K-fold with one constraint: the folds are drawn over group values, not individual windows, so every group lands wholly in one fold and none straddles the train/test boundary. This lets you rotate every entity through the test slice while still measuring generalization to held-out groups, which is decisive when you have few subjects or machines to spend. The splitter partitions the unique group ids into K bins and sends each window to its group's bin. Prefer it over a single GroupShuffleSplit when the group count is small enough that one held-out slice would be too noisy to trust; reserve the single split for when groups are plentiful and extra folds add nothing.

import numpy as np
from sklearn.model_selection import GroupKFold, GroupShuffleSplit

# X: (N, L, C) windows; y: (N,) labels; groups: (N,) subject id per window
rng = np.random.default_rng(0)
N = 2000
groups = rng.integers(0, 20, size=N)      # 20 subjects
y = rng.integers(0, 3, size=N)

# One held-out split: whole subjects go to test, none shared
gss = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=0)
train_idx, test_idx = next(gss.split(np.zeros(N), y, groups))
overlap = set(groups[train_idx]) & set(groups[test_idx])
print("shared subjects:", len(overlap))    # must be 0

# K-fold where every fold is subject-disjoint
for k, (tr, te) in enumerate(GroupKFold(n_splits=5).split(np.zeros(N), y, groups)):
    assert not (set(groups[tr]) & set(groups[te]))
    print(f"fold {k}: {len(set(groups[te]))} test subjects, {len(te)} windows")

Two refinements matter for honest numbers. First, groups are rarely balanced: one subject may contribute ten times the windows of another. A group-disjoint split can then load almost all of one class onto a single side, so use stratified-grouped variants (which preserve each class's proportion within every fold while still keeping groups disjoint) or report per-group performance to catch it. Second, with few groups the estimate swings wildly. Leaving out one of five machines can move the score sharply, so prefer leave-one-group-out cross-validation (rotate the groups so each one takes a turn as the sole test set while every other group trains the model) and report the spread rather than a single fold. This ties directly to the small-sample uncertainty tools from Chapter 4.

When a pure entity split is not enough

When to go further. Group-disjoint splitting removes identity leakage, but two related traps survive it. If your data has a strong temporal structure (a machine degrades over months), you may also need the test period to come after the training period so you do not train on the future, combining a machine split with a temporal cutoff. And if the label is correlated with the group (all positive cases came from one site), even a clean split can hand the model a spurious site-equals-label shortcut. The defense is to check that each class is represented across multiple groups on both sides, and to prefer coarser or composite grouping when a nuisance variable and the label are confounded. These distribution-shift concerns are the running theme of Chapter 66, and the leakage-safe benchmarking discipline is formalized in Chapter 65.

Group-aware splitting is deceptively simple to implement and decisive in effect: the same model and the same data can post a triumphant or a sober number depending only on where you draw the boundary. Draw it at the entity your deployment must generalize across, prove the groups are disjoint, and your accuracy finally means what it says.

What's Next

In Section 5.5, we tackle the nuisance that made device and subject splits necessary in the first place: per-unit variation. Normalization and per-device calibration remove sensor-specific bias and scale so the model learns the signal rather than the serial number; the decisive move is to fit those transforms inside the training fold only, so the cure for one leak does not quietly open another.