Statistical process control

"Every process has a heartbeat. My job is to tell the arrhythmias from the times you simply breathed."

A Level-Headed AI Agent

This section assumes you are comfortable with the Gaussian, standard deviation, and tail probabilities from Chapter 4, and with the ideas of sampling rate and autocorrelation from Chapter 3. We stay with the classical charts here; residual-based thresholds get their own treatment in Section 12.3, and the sequential change-point machinery (CUSUM, BOCPD) in Section 12.4.

Common cause versus special cause

The conceptual heart of SPC, due to Walter Shewhart, is a two-way split of all variation. Common-cause variation is the intrinsic noise floor of a process operating as designed: thermal noise in the ADC, micro-vibration in a bearing, the beat-to-beat jitter of a resting heart. It is stable, predictable in aggregate, and there is no single culprit to chase. Special-cause variation is an intrusion from outside that system: a loosening bolt, a fouled electrode, a firmware regression that biased a reading. A process showing only common-cause variation is said to be in statistical control, meaning its output is stationary and its distribution can be characterized once and trusted.

The distinction matters because it dictates the correct response. Reacting to common-cause noise as if it were a fault, sometimes called tampering, makes things worse: you chase ghosts, adjust settings that were fine, and inflate the very variance you were trying to shrink. Ignoring a special cause lets a real fault ride. SPC is, at bottom, a formal test that runs on every new sample and answers one question: is this still the same process talking, or has something new started to speak?

The Shewhart chart and three-sigma limits

The workhorse is the Shewhart chart. During a clean baseline period (Phase I) you estimate the in-control mean \(\mu_0\) and standard deviation \(\sigma_0\) of the monitored statistic. Then, in monitoring (Phase II), you plot each new value against a center line and a pair of control limits:

The multiplier \(L\) is almost universally set to 3, giving the famous three-sigma limits, and the choice is not arbitrary: for an in-control Gaussian, the probability of a single point falling outside \(\pm 3\sigma\) by chance is about \(0.0027\), so a lone out-of-limit point is strong evidence of a special cause. Shewhart picked three to balance two costs: too tight and you drown in false alarms, too loose and you miss real shifts. That balance is quantified by the average run length (ARL), the expected number of samples until an alarm. In control, \(\text{ARL}_0 = 1/0.0027 \approx 370\): a plain three-sigma chart false-alarms roughly once every 370 samples even when nothing is wrong. Choosing \(L\) is choosing your tolerable false-alarm rate, exactly the threshold-economics tradeoff Section 12.7 makes explicit.

Run rules and the sensitivity dial

A single point outside three sigma is a blunt instrument: it catches large, abrupt shifts but is nearly blind to a small, sustained drift, the kind a slowly degrading sensor produces. The Western Electric rules add pattern detectors that read the sequence, not just the last point: one point beyond \(3\sigma\); two of three consecutive points beyond \(2\sigma\) on the same side; four of five beyond \(1\sigma\) on the same side; and eight consecutive points on one side of the center line, each targeting a signature that pure limit-crossing misses, especially a mean shift too small to escape the outer band on any single sample.

Every rule you add buys sensitivity and spends false-alarm budget. Turn on all four Western Electric rules and the in-control ARL drops from about 370 toward 90, four times the false-alarm rate. There is no free lunch: run rules just trade along the detection-speed-versus-false-alarm curve that governs every detector in this chapter. For a slow drift you would rather catch early, a better tool than piling on rules is a chart with memory.

Memory charts: EWMA for small, persistent shifts

The exponentially weighted moving average (EWMA) chart replaces each raw sample with a running blend of the present and the recent past:

This is exactly the exponential smoother from Chapter 6, now pressed into service as a detector. Because \(y_t\) accumulates evidence across samples, a shift of half a sigma that no single point would flag pushes the smoothed value steadily until it crosses its tightened limits. Small \(\lambda\) (say 0.1 to 0.2) gives a long memory and superb sensitivity to tiny drifts; \(\lambda \to 1\) recovers the memoryless Shewhart chart. The limits shrink because the steady-state variance of \(y_t\) is \(\sigma_0^2\,\lambda/(2-\lambda)\), smaller than \(\sigma_0^2\):

import numpy as np

def ewma_chart(x, mu0, sigma0, lam=0.2, L=3.0):
    y, prev = np.empty(len(x)), mu0
    for t, xt in enumerate(x):
        prev = lam * xt + (1 - lam) * prev      # smoothed statistic
        y[t] = prev
    t_idx = np.arange(1, len(x) + 1)
    spread = np.sqrt(lam / (2 - lam) * (1 - (1 - lam) ** (2 * t_idx)))
    ucl, lcl = mu0 + L * sigma0 * spread, mu0 - L * sigma0 * spread
    alarms = np.where((y > ucl) | (y < lcl))[0]
    return y, ucl, lcl, alarms

rng = np.random.default_rng(0)
baseline = rng.normal(50.0, 1.0, 200)           # Phase I: in control
drift    = rng.normal(50.6, 1.0, 200)           # +0.6 sigma special cause
stream   = np.concatenate([baseline, drift])
y, ucl, lcl, alarms = ewma_chart(stream, mu0=50.0, sigma0=1.0)
print("first alarm at sample", alarms[0] if len(alarms) else "none")

The code above shows the whole mechanism: smooth, compute time-varying limits, flag crossings. Run it and the first alarm lands shortly after sample 200, catching a drift that individual points hide inside the noise.

Bringing SPC to real sensor streams

Classical SPC was designed for occasional, independent quality measurements, and two of its assumptions break on high-rate sensor data. First, autocorrelation: SPC's limits assume independent samples, but a signal at 1 kHz is heavily correlated sample to sample, so the effective sample count feeding a raw Shewhart chart is far smaller than the nominal one, the naive sigma is wrong, and false alarms explode. The fix is a rational subgroup: replace each window of correlated samples, say one second at 1 kHz, with a single summary statistic (its mean or RMS), and chart that sequence instead, since averaging over a window spanning the correlation time washes out the dependence. Size the window near or above that correlation time, found with the autocorrelation tools of Chapter 3: too short and correlation survives, too long and you react slowly. Prefer subgrouping when no dynamic model of the signal exists yet; once one does, charting its residuals, the bridge to Section 12.3, is usually sharper. Second, nonstationarity: diurnal cycles, duty-cycle regimes, and thermal drift violate a single fixed \(\mu_0\). Contextual normalization, per-regime baselines, or a slowly adapting center line keep the chart honest, connecting SPC to the streaming machinery of Chapter 60. Get these two right and a monitor costing microseconds per sample holds its own against models costing a thousand times more.

What's Next

In Section 12.3, we generalize the control-chart idea beyond a single monitored value to residuals: run a model of what the signal should be, watch what it fails to explain, and set thresholds on that error stream. This is how SPC's clean limit logic survives contact with the autocorrelated, nonstationary signals that broke the naive chart, and it sets up the sequential change-point detectors that follow.