Evaluation without complete labels (and the point-adjust trap)

"Give me the right scoring rule and I will make a coin flip look like a genius. That is not a boast. It is a warning about your metric."

A Skeptical AI Agent

This section assumes the precision, recall, and threshold vocabulary from the residual view of Section 12.3, the base-rate reasoning of Chapter 4, and the leakage discipline of Chapter 5. The full benchmarking protocol it feeds into is developed in Chapter 65.

Why complete labels almost never exist

Point-wise anomaly labels for a long sensor stream are a fiction we rarely get to enjoy. Three structural reasons keep them incomplete. First, boundaries are ambiguous: an anomaly is a temporal event with a soft onset and offset, so no two annotators agree on the exact first and last anomalous sample, and asking them to is asking for noise. Second, anomalies are rare: a healthy machine may run for months between events, so the positive class is a fraction of a percent of samples, and a single mislabeled segment swings the score. Third, discovery is retrospective: you learn a fault occurred from a maintenance ticket or a patient outcome, timestamped to the shift or the day, not to the millisecond your detector cares about. The consequence is that we grade detectors against labels that mark events, coarse intervals of "something was wrong here", while the detector emits per-sample decisions. Any honest metric must bridge that granularity gap; how it bridges it is exactly where evaluation goes right or wrong.

The point-adjust trap

Point adjust was introduced with good intentions: if a ground-truth anomaly spans samples \(t_0\) through \(t_1\) and the detector fires on even one sample inside that window, a human would presumably have caught the whole event, so the reasoning goes, credit the detector for the entire segment. Concretely, before scoring you rewrite the prediction: for each labeled anomaly segment, if any point in it is flagged, mark every point in it as flagged, then compute ordinary point-wise precision, recall, and \(F_1\). It sounds humane. In practice it turns segment length into a credit multiplier, and that multiplier is where the trap hides.

Suppose the labeled anomalies occupy contiguous runs averaging \(L\) samples each, and a detector flags points independently with probability \(p\). The chance it touches a given segment, and therefore harvests all \(L\) points of true-positive credit, is

For \(L = 100\) and a trivial flag rate of \(p = 0.02\), that probability is already about \(0.87\). A detector that flags two percent of samples uniformly at random, with zero signal, sweeps up almost all of the recall while its false positives stay diffuse and cheap. The recovered recall is near one, precision looks respectable because the denominator is dominated by the credited segment points, and the resulting point-adjust \(F_1\) rivals or beats genuinely engineered detectors. This is not hypothetical: the finding that a randomized score can outperform state-of-the-art models under point adjust is what forced the field to abandon the protocol for headline numbers. The code below reproduces the effect in a dozen lines.

import numpy as np

def f1(pred, label):
    tp = np.sum(pred & label); fp = np.sum(pred & ~label); fn = np.sum(~pred & label)
    p = tp / (tp + fp + 1e-9); r = tp / (tp + fn + 1e-9)
    return 2 * p * r / (p + r + 1e-9)

def point_adjust(pred, label):          # the trap: any hit credits the whole segment
    pred = pred.copy(); i = 0
    while i < len(label):
        if label[i]:
            j = i
            while j < len(label) and label[j]: j += 1
            if pred[i:j].any(): pred[i:j] = True   # inflate to full segment
            i = j
        else:
            i += 1
    return pred

rng = np.random.default_rng(0)
n = 20000
label = np.zeros(n, dtype=bool)
for s in rng.integers(0, n - 100, size=25):   # 25 anomaly events, ~100 samples each
    label[s:s + 100] = True

pred = rng.random(n) < 0.02              # a PURELY RANDOM detector, no signal at all
print("point-wise  F1 :", round(f1(pred, label), 3))
print("point-adjust F1:", round(f1(point_adjust(pred, label), label), 3))

Metrics that survive incomplete labels

If segment credit is the right instinct but full-segment credit is the trap, the fix is to make credit proportional and localized. Several families do this correctly. Range-based precision and recall (Tatbul and colleagues) generalize the point definitions to intervals, splitting recall into an existence term (did you detect the event at all) and an overlap term (how much of it), with a tunable bias for early or positional detection. Affiliation metrics (Huet and colleagues) score each prediction by its temporal distance to the nearest true anomaly and normalize against what a random baseline would achieve, so a coin flip scores zero by construction. For threshold-free comparison, the volume under the surface (VUS) family generalizes ROC and precision-recall area to anomaly ranges with a tolerance buffer, removing the arbitrary threshold and the razor-edge boundary sensitivity at once.

Two workhorse cautions apply to the classic curves you will reach for first. Prefer the area under the precision-recall curve over the area under the ROC curve, because ROC is seduced by the enormous true-negative count under extreme class imbalance and can look excellent while precision is dismal. And never report "best \(F_1\)", the \(F_1\) at the single threshold that maximizes it on the test set, as your result: choosing the threshold with the answer key in hand is threshold leakage, an optimistic peek that will not survive contact with a production stream where you must fix the threshold in advance. Set the threshold on a validation split or from a false-alarm budget, exactly as in Section 12.3, then report performance at that threshold.

Evaluating with few labels, or none

When labels are scarce you still have levers. The most reliable is controlled fault injection: take verified-healthy data and synthesize anomalies with known onset, offset, and magnitude (spikes, level shifts, variance bursts, frozen sensors), then score against a label you authored precisely. This is the leakage-safe evaluation thread of this book applied to detection; the synthetic-data machinery is developed further in Chapter 55. Complement it with label-free operating characteristics: on healthy-only data you can measure the empirical false-alarm rate and mean time between false alarms directly, no anomaly labels required, which pins down half of the tradeoff by itself. For the detection half, weak or delayed labels (a maintenance ticket dated to the day) support event-level recall and detection delay: samples from a segment's true onset to its first correctly flagged sample. Delay matters because the cost of a fault accrues during exactly that unflagged window, so two detectors with identical event-level recall can let very different amounts of damage through before anyone notices. Compute it as the median across detected events, not the mean, since one event the detector never catches would otherwise blow up the average, and adopt an explicit rule (for example, capping delay at the segment length) for events with no detection at all. Lead with delay whenever an alarm drives a real-time response; lean on the overlap-based scores above when the downstream use is offline review instead. Finally, when scores carry calibrated uncertainty you can evaluate coverage rather than hard hits, connecting to the conformal machinery of Chapter 18.

What's Next

In Section 12.7, we turn the evaluation lens toward the human on the other end of the alarm. A detector with a beautiful precision-recall curve can still be operationally useless if it pages an engineer every ten minutes, so we make the false-alarm budget a first-class economic quantity: how alert fatigue is measured, how a threshold trades detection against operator trust, and how to price an alarm so the system stays worth listening to.