Clinical labels and reference standards

"Everyone kept telling me to learn the ground truth. Nobody warned me the ground was two cardiologists disagreeing at 3 a.m."

A Recently Humbled AI Agent

Formally, we want to estimate a hidden clinical state \(s\) (this beat is a ventricular ectopic; this 30-second epoch is REM sleep; this patient has atrial fibrillation) from a biosignal window. Supervised learning needs a label \(y\) that stands in for \(s\). The reference standard is the procedure that produces \(y\). It is a measurement in its own right, subject to the same \(y = s + \text{error}\) logic as the sensor itself (Chapter 4). Mistaking the label for the state does not remove the label's error, it only hides it, and hidden error is the single most common self-inflicted wound in clinical machine learning.

A hierarchy of reference standards

Not all labels are created equal, and the field has a rough hierarchy. A gold standard is the most definitive practical measurement of the state: polysomnography (PSG) scored under the AASM manual for sleep stage, a synchronized 12-lead ECG read for arrhythmia, an arterial line for blood pressure, a laboratory assay for a biomarker. A silver standard is a defensible proxy: a single-lead patch instead of 12 leads, an oscillometric cuff instead of an arterial line. A weak or surrogate label is cheaper still: a billing code (ICD diagnosis) in the electronic health record, a self-reported symptom, or the output of an older algorithm you are trying to replace. Each step down the hierarchy trades fidelity for volume, and the trade is the central design decision of a biosignal dataset: a million EHR-coded records look more attractive than ten thousand adjudicated strips, until the coding error rate turns out to swamp the effect you are trying to learn.

Labels are noisy: quantify the disagreement

Because a human (or a proxy device) makes the label, the first thing to measure is not model accuracy but label reliability: how much do independent annotators agree? Raw percent agreement is misleading because some agreement happens by chance, especially under class imbalance. Cohen's kappa corrects for chance:

where \(p_o\) is the observed agreement and \(p_e\) is the agreement expected if both raters labeled independently at their observed base rates. \(\kappa = 1\) is perfect agreement, \(\kappa = 0\) is chance-level, and negative values mean systematic disagreement. For more than two raters, Fleiss' kappa generalizes the same idea. The short program below computes Cohen's kappa for two clinicians scoring the same set of ECG strips as normal, ectopic, or noise, and contrasts it with the raw percent agreement that a naive report would quote.

import numpy as np
from sklearn.metrics import cohen_kappa_score

# Two clinicians label the same 12 ECG strips: N=normal, V=ectopic, X=noise
rater_a = np.array(list("NNNNNNNNVVNX"))
rater_b = np.array(list("NNNNNNNNVNNX"))

percent_agree = np.mean(rater_a == rater_b)
kappa = cohen_kappa_score(rater_a, rater_b)

print(f"raw percent agreement: {percent_agree:.2f}")
print(f"Cohen's kappa:         {kappa:.2f}")

The gap between 0.92 and 0.63 in Listing 28.5.1 is the whole lesson: on imbalanced clinical classes, the metric your model will eventually be judged by is dominated by the majority class, so you must report a chance-corrected agreement for the labels themselves before you trust any model number built on them. When agreement is low, the standard remedy is adjudication: multiple annotators score independently, disagreements go to a senior reviewer or a consensus panel, and the resolved label becomes the reference. Alternatively, keep the disagreement as information by training on soft labels (the fraction of raters who chose each class), which pairs naturally with the calibration methods of Chapter 18.

Granularity, timing, and matching conventions

A clinical label also has a shape: it can attach to a single heartbeat, to a variable-length event (an apnea, a seizure, an AFib episode), or to an entire record ("this patient has paroxysmal AFib"). This matters because a record-level label is true if the event occurs even once in hours of recording, so a model can match it while getting the timing, duration, and count of every individual event wrong, and a naive accuracy number will not reveal that. Evaluate at the granularity the label actually encodes: score beats only against beat labels, score record-level predictions as one decision per record, and if you aggregate a finer signal into a coarser prediction (say, flagging a record positive if any epoch inside it is positive), report both levels separately rather than blending them into one figure. The mismatch bites hardest whenever the true event is rare within a long record, exactly the regime Chapter 29 (ECG and Cardiac AI) develops in depth for beat-level versus record-level arrhythmia classification.

For event and beat labels, agreement also depends on timing tolerance: a detected beat at 412 ms and a reference beat at 430 ms should count as a match, not a miss plus a false alarm. The field standardized this. The ANSI/AAMI EC57 standard defines how to score arrhythmia detectors, including a 150 ms matching window around each reference beat, and databases such as MIT-BIH ship their labels in the WFDB annotation format with a documented symbol set (N for normal, V for ventricular ectopic, and so on). Using these conventions is not bureaucracy; it is what makes your reported sensitivity comparable to a published baseline instead of an artifact of your own matching rule. This is the same leakage-and-comparability discipline that Chapter 5 and Chapter 65 insist on for benchmarking.

import wfdb.processing as wp

# ref_samples / test_samples: beat locations in samples; fs = sampling rate
comparator = wp.compare_annotations(ref_samples, test_samples,
                                    window_width=int(0.15 * fs))  # 150 ms EC57 window
print(comparator.sensitivity, comparator.positive_predictivity)

The reference contaminates: leakage and prevalence traps

Two failure modes recur often enough to name. The first is reference leakage. When the label comes from a co-recorded gold-standard device, its signal can bleed into the very features you train on: a chest-belt reference for respiration leaves a synchronization artifact in the wearable's accelerometer, or the reference ECG shares a common ground with the PPG amplifier. The model then learns to detect the presence of the reference wire rather than the physiology, and posts beautiful numbers that collapse the instant the reference is unplugged in deployment. Guard against it by asking, for every feature, whether it could exist only because the reference was attached.

The second is prevalence dependence. Sensitivity and specificity are properties of the detector, but positive predictive value (PPV), the metric a clinician actually experiences as "how often is an alarm real", depends on how common the condition is:

where \(\pi\) is prevalence. A 99%-specific AFib screener sounds excellent until you deploy it on a healthy 40-year-old population where \(\pi\) is well under 1%, and the majority of its alarms become false. This is why the consumer-wearable screening studies of Chapter 30 (PPG and Wearable Cardiovascular Sensing) are engineered around low-prevalence populations, and why the clinical-validation designs of Chapter 34 (Clinical Validation, Regulation, and Biometric Privacy) report PPV at the intended-use prevalence rather than at the enriched prevalence of the training set. The reference standard defined your labels; the deployment prevalence redefines what those labels are worth.

What's Next

In Section 28.6, we turn from how labels are made to how records are split. Even a perfect reference standard is wasted if beats from the same patient land in both the training and test sets, so we build patient-level splits and hunt down the subtle physiological leakage that lets a model recognize the person instead of the pathology.