Relative and relational objectives (RelCon)

"Two joggers are not identical and not strangers. Most of the world lives in that gap, and a binary label refuses to see it."

A Graded AI Agent

Why binary positives break on sensor data

The augmentation-based positive works when a single semantic class dominates a window and augmentations are label-preserving, an assumption vision inherited from crops of one object. Sensor streams violate both halves of it. A ten-second inertial window can contain a gait cycle plus a stumble, and jitter or scaling augmentations sometimes cross a semantic boundary (a small time-warp can turn a slow walk into something the downstream head reads as a shuffle). Worse, the negative set is polluted: in any reasonably sized batch of wearable data, several "negatives" are other subjects doing the same activity, and InfoNCE dutifully repels them, injecting label noise no amount of data cleans up.

The fix is to stop pretending similarity is binary. If we had, for each anchor, a trustworthy ordering of candidates from most to least similar, we could ask the embedding to reproduce that ordering rather than a 0/1 verdict. This turns pretraining into a ranking problem over relationships between distinct real samples, hence "relative" and "relational." The two engineering questions become: (1) where does a reliable similarity ranking come from without labels, and (2) what loss turns a ranking into gradients.

A learnable, label-free similarity measure

RelCon answers question (1) with a learnable distance measure trained before the main encoder. The measure judges how well one motion segment can explain another: given a candidate segment, can it reconstruct the masked portions of the anchor after an alignment that tolerates speed differences? Dynamic time warping supplies the alignment, so two walks at different cadences are recognized as close, and a reconstruction objective supplies the graded score. The result is a function \(d(x, c)\), small when \(c\) is a good motif-level match for anchor \(x\) and large otherwise, learned entirely from unlabeled signals. This is the sensor-native analogue of the intuition in Chapter 7 that similar signals share time-frequency structure, but here "similar" is fit to the data rather than fixed by hand.

For each anchor we sample \(K\) candidates from the corpus and score them with \(d\), yielding an ordering \(d_{(1)} \le d_{(2)} \le \dots \le d_{(K)}\). Nearest candidates are soft positives; farthest are clear negatives; the middle is exactly the graded region binary methods throw away.

The relative contrastive loss

Question (2) is answered by a nested InfoNCE. Order the candidates by learned distance. Then, for every rank \(j\), treat candidate \(j\) as the positive and all candidates at least as far as \(j\) as its negatives. Summing these nested terms gives

where \(z\) is the anchor embedding, \(z_{(k)}\) is the embedding of the \(k\)-th nearest candidate, \(s\) is cosine similarity, and \(\tau\) is the temperature. Each term is an ordinary contrastive loss; stacking them forces \(s(z, z_{(1)}) > s(z, z_{(2)}) > \dots\), so the embedding geometry mirrors the similarity ranking. A single candidate is simultaneously a positive (against those farther) and a negative (against those nearer), which is precisely the relational bookkeeping a binary loss cannot express.

import torch
import torch.nn.functional as F

def relative_contrastive_loss(z_anchor, z_cands, dist, tau=0.1):
    # z_anchor: (D,)  z_cands: (K, D)  dist: (K,) learned distances to anchor
    z_anchor = F.normalize(z_anchor, dim=-1)
    z_cands  = F.normalize(z_cands, dim=-1)
    sim = (z_cands @ z_anchor) / tau            # (K,) cosine sim, scaled
    order = torch.argsort(dist)                 # nearest (smallest dist) first
    sim = sim[order]
    losses = []
    for j in range(sim.shape[0] - 1):
        # candidate j is positive; all farther candidates (k >= j) are negatives
        denom = torch.logsumexp(sim[j:], dim=0)
        losses.append(denom - sim[j])
    return torch.stack(losses).mean()

# toy check: distances that agree with embedding sims give a lower loss
z  = torch.randn(64)
zc = torch.randn(8, 64)
d  = torch.arange(8.0)                          # candidate 0 is "closest"
print(float(relative_contrastive_loss(z, zc, d)))

The loop above is written for clarity; a production implementation vectorizes it across a batch of anchors, as the callout below explains. The temperature and the candidate count \(K\) are the two knobs that matter: too small a \(K\) and the ranking is noisy, too large and most candidates are trivially far and contribute nothing.

When relational objectives are the right tool

Reach for a relative or relational loss when three conditions hold: labels are absent, the corpus has heavy class overlap (many subjects or devices repeating the same motions, so false negatives are rife), and you can define a cheap, trustworthy similarity between distinct samples. Inertial and physiological streams (Chapter 23) fit perfectly. When the corpus is genuinely single-class per window and augmentations are safe, the extra machinery buys little and plain SimCLR-style contrastive learning is simpler and just as good.

Nearest-neighbor variants such as NNCLR (Dwibedi et al., 2021) are a lighter-weight cousin, useful when a bespoke similarity measure is not worth engineering. Instead of ranking a whole candidate set against a learned distance, NNCLR keeps a support queue of recent embeddings and, at each step, swaps the augmented view for the anchor's single nearest neighbor in that queue, then runs ordinary binary InfoNCE against it: a real, different sample stands in for the synthetic positive, at the cost of a queue lookup rather than a trained distance model and a nested loss. What it gives up is the graded ordering that makes RelCon's geometry finer-grained, so reach for the full ranking when the downstream task rewards that, such as regressing energy expenditure rather than only classifying activity. As always in this book, evaluate on a leakage-safe split: because relational objectives explicitly pull same-activity samples together, a subject leaking across the train/test boundary can inflate scores dramatically, so hold out whole subjects and devices per Chapter 5.

What's Next

In Section 17.6, we scale the relational idea across boundaries: pretraining that spans different devices, different users, and different modalities at once, where the notion of "similar" must survive a change of sensor, sampling rate, and body location.