Statistical, shape, and entropy features

"The mean tells me where the signal lives. The kurtosis tells me whether it is hiding a knife behind its back."

A Distribution-Minded AI Agent

This section leans on the probability and estimation background from Chapter 4: moments, quantiles, and the idea that a finite window is a noisy estimator of an underlying distribution. Everything here is computed on the same fixed analysis window you have used throughout the chapter.

Statistical moments: the distribution as a fingerprint

Treat the \(N\) samples in a window as draws from a distribution and summarize that distribution with its moments. The first two are familiar: the mean \(\mu\) and the variance \(\sigma^2\). The interesting discrimination usually lives in the standardized third and fourth moments. Skewness measures asymmetry, and kurtosis measures how heavy the tails are relative to a Gaussian:

Why do these matter for sensors? Because a great many fault and event signatures are impulsive: they leave the bulk of the signal unchanged but inject rare, large excursions. Kurtosis is exquisitely sensitive to exactly those rare large values, since the deviations are raised to the fourth power. A healthy rotating machine produces near-Gaussian vibration with kurtosis close to 3; the first pits on a bearing race add sharp periodic impacts that push kurtosis to 5, 8, or higher long before the RMS energy moves. Skewness catches asymmetry: a photoplethysmography pulse has a steep upstroke and a slow decay, so its skew is a stable shape marker used in the wearable cardiovascular work of Chapter 30. Beyond moments, distribution-free quantiles (median, interquartile range, the 5th and 95th percentiles) give robust location and spread that a single outlier cannot wreck, which is why they pair so well with the heavy-tailed moments.

Shape features: the geometry of the waveform

Shape features describe the form of the waveform or of its value histogram, often as dimensionless ratios that survive changes in sensor gain and units. The crest factor, the ratio of peak amplitude to RMS, quantifies how "peaky" a signal is: a pure sine sits at \(\sqrt{2}\approx 1.41\), while an impulsive fault signal climbs well above it. Related dimensionless ratios (the shape factor, impulse factor, and clearance factor) are staples of vibration diagnostics because they cancel out the operating amplitude and isolate waveform form:

Other shape descriptors work on the trace itself: peak-to-peak span, the number and prominence of local maxima, curve arc length as a roughness proxy, and the area under the rectified signal. Biosignals lean hard on these: an electrocardiogram beat is described by the width, amplitude, and slope of its QRS complex, features that carry straight into the cardiac models of Chapter 29. Shape features encode what a cycle looks like, not how loud it is: exactly the invariance a robust classifier wants.

Entropy and complexity: how predictable is the signal?

Entropy features answer a question the moments cannot: given the recent past, how surprised are you by the next sample? A perfectly periodic signal is maximally predictable and scores low entropy; broadband noise is maximally surprising and scores high; and, importantly, many physiological and mechanical systems lose complexity as they fail or age, landing somewhere in between. Several definitions matter in practice. Spectral entropy treats the normalized power spectrum as a probability distribution and computes its Shannon entropy, measuring whether energy is concentrated in a few tones (low entropy) or spread across the band (high entropy). Sample entropy and approximate entropy measure temporal regularity by asking how often short patterns of length \(m\) that match within a tolerance \(r\) still match when extended to length \(m+1\): a signal whose short patterns keep matching as you extend them is regular and scores low, while one whose matches keep breaking is complex and scores high. Approximate entropy counts a pattern as matching itself, biasing short records toward looking more regular than they are; sample entropy fixes this by excluding self-matches and taking the log of the ratio of match counts at lengths \(m\) and \(m+1\), which is why it has largely replaced approximate entropy in practice. Both need hundreds to thousands of samples to populate the match counts reliably, with \(m=2\) and \(r \approx 0.2\sigma\) as workable defaults, and both earn their keep over permutation entropy specifically when magnitude, not just order, carries diagnostic information, as in heart-rate variability analysis where \(r\) is tuned to the physiological noise floor. Permutation entropy is the cheap, robust favorite: it looks at the ordinal pattern (the rank order) of each short window of samples and computes the Shannon entropy of how often each pattern appears.

For a length-\(m\) ordinal pattern \(\pi\) occurring with relative frequency \(p(\pi)\), permutation entropy is simply

normalized by \(\log(m!)\) to land in \([0,1]\). Because it depends only on the order of values and not their magnitudes, permutation entropy is nearly immune to sensor drift, slow baseline wander, and monotone gain changes, the failure modes that plague raw amplitude features. Entropy features are the workhorses of the anomaly and change detectors in Chapter 12, where a sudden drop in regularity is the first sign that a monitored system has left its normal regime.

import numpy as np
from scipy.stats import skew, kurtosis
from math import factorial, log
from itertools import permutations

def permutation_entropy(x, m=3, tau=1):
    """Normalized permutation entropy of a 1-D signal."""
    patterns = {}
    for i in range(len(x) - (m - 1) * tau):
        window = x[i:i + m * tau:tau]
        order = tuple(np.argsort(window))     # the ordinal pattern
        patterns[order] = patterns.get(order, 0) + 1
    counts = np.array(list(patterns.values()), dtype=float)
    p = counts / counts.sum()
    H = -(p * np.log(p)).sum()
    return H / log(factorial(m))              # in [0, 1]

fs = 2000
t = np.arange(0, 1.0, 1 / fs)
healthy = np.sin(2 * np.pi * 50 * t) + 0.05 * np.random.randn(t.size)
impacts = 3.0 * (np.random.rand(t.size) < 0.01)      # rare sharp spikes
faulty  = healthy + impacts

for name, sig in [("healthy", healthy), ("faulty", faulty)]:
    print(f"{name:8s} kurtosis={kurtosis(sig)+3:5.2f} "
          f"skew={skew(sig):+5.2f} Hperm={permutation_entropy(sig):.3f}")

The code above makes the point concrete: kurtosis catches the spike, permutation entropy catches the broken rhythm, and where the two disagree is usually the most informative signal of all, which is exactly why you compute several complementary features rather than betting on one.

import antropy as ant
h_perm = ant.perm_entropy(faulty, normalize=True)
h_spec = ant.spectral_entropy(faulty, sf=fs, method="welch", normalize=True)
h_samp = ant.sample_entropy(faulty)

Computing them without self-deception

Three cautions keep these features from lying. First, moments and entropies are estimators: on a short window, kurtosis and sample entropy have large variance, so a single window can look alarming purely by chance; aggregate over several windows or lengthen the window when the phenomenon allows. Second, scale sensitivity cuts both ways: amplitude-based features (variance, peak-to-peak) demand that you fix calibration and units, while ordinal features (permutation entropy) and dimensionless ratios (crest factor) shrug off gain changes, so choose deliberately based on whether your sensor's gain is trustworthy. Third, and most important for machine learning, these features are computed per window and fed to a model, so normalization statistics (feature means and standard deviations) must be fit on the training split only; fitting a scaler on the whole dataset leaks test information and inflates accuracy, exactly the trap the leakage-safe pipeline of Chapter 5 is built to prevent.

What's Next

In Section 8.4, we stop computing features one function at a time and reach for curated libraries. Tools like tsfresh and catch22 compute hundreds of statistical, shape, spectral, and entropy features in a single call, complete with automated relevance filtering, turning the hand-built columns of this section into an industrial feature factory.