Calibration and temperature scaling for time series

"When I say I am 90 percent sure, I want to be wrong exactly one time in ten. A confident liar is worse than an honest doubter."

A Well-Calibrated AI Agent

What calibration means, and how to measure it

Let a classifier output a probability vector \(\hat{p}(x)\) with predicted class \(\hat{y}=\arg\max_k \hat{p}_k(x)\) and confidence \(\hat{c}=\max_k \hat{p}_k(x)\). Perfect calibration is the statement

$$\mathbb{P}\big(\hat{y}=y \,\big|\, \hat{c}=c\big) = c \quad\text{for all } c\in[0,1].$$

Read literally: among all windows the model called "80 percent," it should be right 80 percent of the time. We cannot condition on a continuous \(c\), so we bin. Sort predictions into \(M\) confidence bins \(B_1,\dots,B_M\), and compare each bin's average confidence to its empirical accuracy. The Expected Calibration Error is the weighted average gap,

$$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n}\,\big|\operatorname{acc}(B_m) - \operatorname{conf}(B_m)\big|,$$

and plotting \(\operatorname{acc}(B_m)\) against \(\operatorname{conf}(B_m)\) gives the reliability diagram: the diagonal is perfect, points below it are overconfident (the usual failure), points above underconfident. ECE is a diagnostic, not a training loss, and can be gamed; report it alongside negative log-likelihood or Brier score, which are proper scoring rules (Chapter 4), so improving calibration cannot secretly wreck sharpness.

Temperature scaling: one number, learned after training

Train the network normally. Freeze it. Now introduce a single scalar \(T>0\), the temperature, and soften every logit vector \(z(x)\) before the softmax:

$$\hat{p}_k(x;T) = \frac{\exp\big(z_k(x)/T\big)}{\sum_j \exp\big(z_j(x)/T\big)}.$$

With \(T=1\) nothing changes. \(T>1\) flattens the distribution toward uniform (less confident); \(T<1\) sharpens it. Because \(T\) scales all logits equally, the argmax and every predicted label is invariant: temperature scaling can change how loudly the model speaks, never what it says. Fit \(T\) by minimizing negative log-likelihood on a held-out calibration set the network never trained on, a one-dimensional, convex-in-practice optimization solved in a handful of iterations. This is Guo et al.'s 2017 result: a single temperature recovers near-perfect calibration across modern architectures, outperforming heavier fixes like Platt scaling, isotonic regression, or Bayesian binning, while adding exactly one parameter. The code below fits \(T\) on cached logits.

import numpy as np
from scipy.optimize import minimize_scalar

def temperature_scale(logits, labels):
    # logits: (n, K) raw pre-softmax outputs on the CALIBRATION split
    # labels: (n,) integer class ids
    def nll(T):
        z = logits / T
        z = z - z.max(axis=1, keepdims=True)          # stable softmax
        logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
        return -logp[np.arange(len(labels)), labels].mean()
    res = minimize_scalar(nll, bounds=(0.05, 10.0), method="bounded")
    return res.x

def ece(probs, labels, n_bins=15):
    conf = probs.max(axis=1); pred = probs.argmax(axis=1)
    correct = (pred == labels).astype(float)
    bins = np.linspace(0, 1, n_bins + 1)
    e = 0.0
    for lo, hi in zip(bins[:-1], bins[1:]):
        m = (conf > lo) & (conf <= hi)
        if m.any():
            e += m.mean() * abs(correct[m].mean() - conf[m].mean())
    return e

T = temperature_scale(cal_logits, cal_labels)         # e.g. T = 1.9
scaled = np.exp(cal_logits / T)
scaled /= scaled.sum(axis=1, keepdims=True)
print(f"T={T:.2f}  ECE {ece(softmax(test_logits), test_labels):.3f} "
      f"-> {ece(softmax(test_logits / T), test_labels):.3f}")

Why time series breaks the textbook recipe

Every guarantee above rests on one assumption: the calibration split is a representative, exchangeable sample of what test time looks like. For a shuffled image benchmark that is nearly free. For a sensor stream it is where projects fail silently. Three specific traps:

• Autocorrelation inflates your effective calibration set. Ten thousand consecutive accelerometer windows from one afternoon are not ten thousand independent samples; they may be worth a few dozen, so a temperature fit on them is fit to that afternoon's operating point and its ECE estimate is optimistically tiny. Calibrate on windows spanning many sessions, subjects, and devices, and evaluate with a block or subject-wise split, never a random per-window shuffle (the leakage rule of Chapter 5).
• Drift makes a fixed \(T\) stale. A sensor that ages, a patient whose baseline shifts, a machine entering a new load regime: the logit distribution moves, and the temperature honest in March is overconfident by September. Temperature scaling is a static, one-shot repair; under genuine distribution shift you need the adaptive machinery of Chapter 66 and the online conformal methods of Section 18.5.
• Step-level versus sequence-level. A model emitting a probability every 20 ms produces thousands of dependent confidences per event; pooling them into one ECE estimate treats a single fall or arrhythmia episode as hundreds of independent trials it never was, the same autocorrelation inflation as above but at the event level rather than the session level. The fix is to aggregate first: reduce each event's per-step probabilities to one number, a max, a mean, or a small learned pooling head, then fit and evaluate \(T\) on that aggregate. Do this whenever one physical occurrence spans multiple inference steps, the norm for windowed sensor classifiers; only a genuinely single-shot decision, one classification per non-overlapping window, needs no aggregation.

Calibrating regression and forecast intervals

Most sensor deep learning is regression or forecasting, not classification, and "calibration" generalizes cleanly. A probabilistic forecaster emits quantiles or a predictive distribution; it is calibrated if its stated coverage matches reality. Concretely, for every level \(\alpha\), the fraction of test targets falling below the predicted \(\alpha\)-quantile should equal \(\alpha\), and a 90 percent prediction interval should contain the truth 90 percent of the time. The regression analogue of the reliability diagram plots nominal coverage against empirical coverage; the analogue of temperature scaling is a learned rescaling of the predicted standard deviation \(\hat{\sigma}\to s\,\hat{\sigma}\), or a monotone recalibration map fit on held-out residuals (Kuleshov et al., 2018). The same three traps apply, sharpened: interval calibration under autocorrelation and drift is exactly the pressure that motivates conformal prediction.

What's Next

In Section 18.3, we stop repairing a single point estimate after the fact and instead ask the model to represent its own uncertainty from the start: deep ensembles, MC-dropout, Bayesian layers, and evidential deep learning. Temperature scaling fixed the confidence you already had; those methods try to produce a trustworthy distribution in the first place, and they capture the epistemic gap that a lone temperature, however honest, cannot see.