Section 7.2: Spectrograms and the STFT

"A single spectrum tells me what frequencies exist. A spectrogram tells me when they showed up to work, and when they quietly clocked out."
A Time-Aware AI Agent

The Big Picture

The Fourier transform from Section 7.1 answers "which frequencies?" but throws away "when?" For a stationary hum that is fine. Real sensor signals are almost never stationary: an engine revs, a bearing starts to click intermittently, a heartbeat speeds up, a gesture starts and stops. The Short-Time Fourier Transform (STFT) recovers the missing time axis by chopping the signal into short frames, transforming each one, and stacking the results into a spectrogram: in effect, a movie of spectra, one frame at a time. That image is the single most reused representation in sensory AI: it is the front end for cough detectors on wearables, engine-order tracking in vehicles, and the input tensor for the audio and vibration convolutional networks you will meet in Chapter 13. This section shows how to build one, how to read one, and how to choose its parameters so it does not lie to you.

This section assumes you are comfortable with sampling and the Nyquist rate from Chapter 3 and with windowing and spectral leakage from Section 7.1. Everything here builds directly on the single-frame DFT; the STFT is simply that DFT applied on a moving window.

From one transform to a moving one

The STFT slides a window $w[n]$ of length $L$ along the signal $x[n]$, advancing by a hop $H$ samples between frames, and computes a DFT of each windowed segment. For frame index $m$ and frequency bin $k$:

$$X[m,k] = \sum_{n=0}^{L-1} x[n + mH]\, w[n]\, e^{-j 2\pi k n / L}.$$

Three knobs fully determine the transform. The window length $L$ sets how much signal each frame sees. The hop $H$ sets how far the window jumps, so the frame overlap is $1 - H/L$. The window shape $w[n]$ (Hann, Hamming, Gaussian) controls leakage exactly as in a single-frame FFT. The spectrogram is the magnitude squared, $S[m,k] = |X[m,k]|^2$, almost always viewed in decibels, $10\log_{10} S$, because sensor energy spans many orders of magnitude and the log compresses it into something the eye and a network can both use.

Why overlap at all? A window tapers to zero at its edges, so events near a frame boundary are attenuated. Overlapping frames (typically 50% to 75%, that is $H = L/2$ to $L/4$) guarantee every sample is seen near the center of at least one window. The formal requirement for a spectrogram you can invert back to a waveform is the Constant Overlap-Add (COLA) condition: the shifted windows must sum to a constant. A Hann window with 50% overlap satisfies COLA, which is why it is the default almost everywhere.

Key Insight: You Cannot Beat the Uncertainty Principle

Time resolution and frequency resolution trade off against each other, and the product is bounded. A frame of length $L$ at sample rate $f_s$ spans $L/f_s$ seconds and resolves frequencies no finer than $\Delta f \approx f_s / L$ hertz. Their product $\Delta t \cdot \Delta f \approx 1$ is fixed: this is the Gabor limit, the signal-processing sibling of Heisenberg's principle. A short window pins down when a transient happened but smears which frequency it was; a long window resolves closely spaced tones but blurs the moment they appeared. There is no window that is sharp in both. Every spectrogram you make is a deliberate choice on this tradeoff, so choose it to match the phenomenon you are hunting: short windows for clicks and impacts, long windows for slowly drifting tones.

Choosing parameters so the picture is honest

Start from the physics, not from a default. Ask what the fastest event you care about is and what the closest two frequencies you must separate are. If a bearing impact lasts about 2 ms, a 256 ms window will paint it as a vertical smear with no useful frequency content; you want a window closer to a few milliseconds. If instead you must distinguish a 49.5 Hz line-frequency tone from a 50 Hz one, you need $\Delta f \le 0.5$ Hz, hence $L \ge f_s / 0.5$ samples, a window of at least two seconds. These two goals can be flatly incompatible on the same signal, which is exactly why the same chapter also offers wavelets (Section 7.3) that adapt resolution across the band.

Two further points prevent self-deception. First, zero-padding each frame, appending zeros before the FFT so its length $N$ grows past the window length $L$, shrinks the plotted bin spacing to $f_s/N$ and makes the curve look smoother, but it only interpolates the same underlying spectrum: the window sets the true resolution $\Delta f \approx f_s/L$, so two tones it cannot separate stay merged no matter how finely you resample between them. It is useful for pinning down one peak's frequency by interpolation, never a substitute for a longer window when the task is separating two close tones. Second, the vertical axis only reaches $f_s/2$; anything above Nyquist has already aliased into the band before the STFT ever ran, so a clean-looking spectrogram can still be built on corrupt data if the anti-alias filtering from Chapter 6 was skipped.

Misconception: Zero-Padding Is Not Free Resolution

It is tempting to raise the FFT length (say from 256 to 4096) and read the crisper-looking plot as proof of finer frequency resolution. The extra bins are computed from the same $L$ real samples; they add interpolated points between the resolution limit the window already fixed, not new information about the signal. If two tones sit closer together than $f_s/L$, no amount of zero-padding pulls them apart on the spectrogram; only a longer window, traded against time resolution per the Gabor limit, can. Use zero-padding to read off a peak's location more smoothly; use a longer window when the job is separating two close frequencies.

import numpy as np
from scipy.signal import ShortTimeFFT
from scipy.signal.windows import hann

fs = 4000                       # sample rate, Hz
t = np.arange(0, 2.0, 1/fs)     # two seconds

# A tone that sweeps 200 -> 800 Hz, plus a short impact at t = 1.2 s
sweep = np.sin(2*np.pi*(200 + 300*t)*t)
impact = np.exp(-((t - 1.2)/0.004)**2) * np.sin(2*np.pi*1500*t)
x = sweep + 0.8*impact + 0.05*np.random.randn(t.size)

L, hop = 256, 64                          # 64 ms window, 75% overlap
win = hann(L, sym=False)                  # COLA-friendly taper
SFT = ShortTimeFFT(win, hop=hop, fs=fs, scale_to="magnitude")
Sx = SFT.stft(x)                          # complex, shape (freqs, frames)

power_db = 20*np.log10(np.abs(Sx) + 1e-9)
freqs = SFT.f                             # Hz axis
times = SFT.t(x.size)                     # seconds axis
print(power_db.shape, freqs[-1], round(fs/L, 1), "Hz per bin")

Building a magnitude spectrogram of a frequency sweep plus a localized impact with SciPy's ShortTimeFFT. The 256-sample Hann window gives $\Delta f \approx 15.6$ Hz and $\Delta t \approx 64$ ms; the sweep appears as a rising ridge and the impact as a bright vertical streak near 1.2 s. Shorten L to sharpen the impact and watch the ridge blur.

The code above makes the tradeoff tangible: run it, then set L = 64 and re-run. The impact snaps into a crisp column while the sweep's ridge fattens vertically. That single experiment teaches more about the Gabor limit than any inequality.

Practical Example: Catching a Cough on a Wrist Wearable

A respiratory-health team built a cough detector for a wrist device with a low-power microphone sampled at 4 kHz. A raw waveform classifier drained the battery and overfit to room noise. Switching the front end to a log-mel spectrogram (a spectrogram whose frequency axis is warped to the perceptual mel scale and whose bands are pooled) changed the economics. They used a 64 ms Hann window with 75% overlap, enough time resolution to separate the explosive onset of a cough from the trailing wheeze, and enough frequency resolution to place its energy in the 300 Hz to 1 kHz band where coughs concentrate. The 2D image fed a tiny convolutional network. Two facts made it ship: the spectrogram is cheap to compute in fixed point on the device, and it discards phase, so trivial time shifts of the same cough map to nearly identical images, giving the classifier built-in shift tolerance. The same log-mel front end underlies the audio branch of many clinical monitors in Chapter 33.

Reading a spectrogram like an instrument

Once you can build one, learn to read it. Steady tones are horizontal lines; a machine running at constant speed shows a fundamental plus evenly spaced harmonics stacked above it. A rising or falling speed bends those lines into sloped ridges, the basis of engine-order tracking in vehicles. Broadband transients (impacts, footsteps, switch clicks) are vertical streaks that span many frequencies at one instant. Amplitude modulation, the hallmark of a developing bearing fault, shows as sidebands: faint lines flanking a strong carrier, spaced by the modulation rate. Recognizing these four patterns (lines, ridges, streaks, sidebands) turns the spectrogram from a pretty picture into a diagnostic readout, and it is the visual vocabulary that the envelope and cepstral methods of Section 7.5 then quantify.

Right Tool: Let the Library Handle the Bookkeeping

A from-scratch, correct STFT plus mel warping plus dB scaling plus proper axis vectors is roughly 40 to 60 lines once you handle edge padding, COLA normalization, and the mel filterbank. With librosa the log-mel spectrogram most audio and vibration pipelines actually use is three lines:

import librosa
mel = librosa.feature.melspectrogram(y=x, sr=fs, n_fft=256, hop_length=64, n_mels=40)
mel_db = librosa.power_to_db(mel, ref=np.max)

The library owns the framing, the Hann window, the mel filterbank construction, and the numerically safe log. You still own the one thing no library can guess: whether 256 samples is the right window for your phenomenon. Choose that yourself; delegate the rest.

A production-grade log-mel front end in three lines with librosa, replacing roughly 40 to 60 lines of hand-written STFT, filterbank, and scaling code.

Research Frontier: Learning the Window Instead of Guessing It

Choosing $L$, $H$, and the window shape by hand, as this section teaches, still dominates production pipelines, but active research treats those choices as parameters to learn rather than tune. Differentiable STFT layers and learnable filterbanks such as SincNet and LEAF let gradient descent adjust window length and center frequencies jointly with the downstream classifier, sometimes finding a per-task tradeoff no human would have guessed. Whether a learned front end transfers across sensors and deployment sites as reliably as a physics-derived one remains open, and it matters directly for the self-supervised pretraining strategies of Chapter 17.

Exercise

Take the synthetic signal from the code block. (1) Compute spectrograms at $L = 512, 256, 64$ with 75% overlap and describe how the sweep ridge and the impact streak each change. (2) For each $L$, report $\Delta t$ and $\Delta f$ and confirm their product stays near 1. (3) Add a second tone at 210 Hz and find the smallest $L$ that visibly separates it from the 200 Hz sweep onset. Explain your answer in terms of $\Delta f$.

Self-Check

You need to distinguish two vibration tones 3 Hz apart at $f_s = 6$ kHz. What minimum window length do you need, and what time resolution does that force?
Why does 50% overlap with a Hann window matter for reconstructing the waveform, and what condition is it satisfying?
A colleague zero-pads every frame from 256 to 4096 samples and claims better frequency resolution. Are they right? What did they actually gain?

What's Next

In Section 7.3, we lift the fixed-window constraint that the Gabor limit imposes on the STFT. Wavelets and multiresolution analysis use short windows at high frequencies and long windows at low ones, giving sharp timing for transients and fine frequency detail for slow trends in the same transform, exactly the flexibility a single spectrogram cannot offer.