What This Book Covers | Building Sensory AI

One thesis, six verbs

Every sensor tells the same kind of story. A photodiode, an accelerometer, a microphone, a radar antenna, and a strain gauge in a bridge deck all report numbers that are corrupted, incomplete, and only indirectly related to the thing you actually care about. The temperature you want is not the voltage you measure; the fall you want to detect is not the raw triaxial acceleration; the machine fault you want to catch is not the vibration spectrum by itself. Sensory intelligence is the discipline of recovering hidden state from noisy physical measurements and then acting on that estimate reliably enough to matter. Formally, a sensor gives you an observation \(y = h(x) + \varepsilon\) generated by a latent state \(x\) you never see directly, and the whole game is producing a good posterior \(p(x \mid y)\) and a decision that respects its uncertainty.

The book organizes that game around six verbs. Every technique you meet is an instance of one of them.

Verb	What it does
Measure	Understand the transducer physics that turns a physical quantity into a number, and the noise, bias, and sampling limits baked in at the source.
Clean	Remove artifacts, drift, and interference; resample, denoise, and calibrate so the signal reflects the world rather than the instrument.
Represent	Turn raw streams into features, spectra, embeddings, or learned latents that expose the structure a downstream model can use.
Infer	Estimate the hidden state: classify, regress, track, detect anomalies, and quantify how confident the estimate is.
Fuse	Combine multiple sensors, times, and physical models into a single coherent world estimate that beats any one channel.
Deploy	Ship the system to real hardware under latency, power, drift, safety, and evaluation constraints, and keep it trustworthy in the field.

The fourteen-part arc

The fourteen parts walk those verbs from first principles to the frontier. Each builds on the last, though the modality parts also stand alone as references.

Foundations and sensor physics. Sampling, quantization, noise models, calibration, and what a measurement actually is.
Classical signal processing and state estimation. Filtering, spectral and wavelet analysis, and the Kalman family that still underpins production systems.
Deep learning and foundation models for sensor time series. Convolutional, recurrent, and transformer architectures, self-supervision, and pretrained models adapted to physical streams.
Agentic sensing. Perception loops that plan measurements, control sensors actively, and reason about what to sample next.
Motion and inertial. Accelerometers, gyroscopes, magnetometers, dead reckoning, activity recognition, and pose.
Biosignals and wearables. ECG, PPG, EEG, EMG, and the physiology and artifacts that make them hard.
Industrial and infrastructure. Vibration, acoustic emission, condition monitoring, predictive maintenance, and structural health.
Radar, lidar, depth, thermal, event, and RF. The active and exotic modalities that see beyond the visible.
Fusion and world models. Multisensor integration, probabilistic state estimation across channels, and learned models of the environment.
Tactile and embodied. Touch, force, proprioception, and sensing inside a body that acts on the world.
Edge and TinyML. Quantization, pruning, on-device inference, and running perception in kilobytes and milliwatts.
Trust, safety, and evaluation. Uncertainty, calibration, robustness, distribution shift, and honest benchmarking.
Applications and case studies. End-to-end systems that combine several parts into a working product.
Frontier. Open problems, emerging modalities, and where machine perception is heading.

The full modality spectrum, built twice

The book covers the whole span of physical sensing: inertial and motion, acoustic and ultrasonic, optical and thermal, radar and lidar and RF, electrophysiological, chemical and environmental, tactile and force, and event-based vision. The unifying claim is that the six verbs transfer across all of them; an anomaly detector for bearing vibration and one for an ECG trace share far more than they differ.

Every core idea appears twice. First it is built from scratch in plain code, so you see exactly what the math does. Then the same idea is shown with the modern library a practitioner would actually reach for, so you can move fast without mistaking the abstraction for magic. A smoothing filter, for instance, is written directly as a convolution before it is called in one line.

import numpy as np

def moving_average(y, k):
    # from scratch: a length-k box filter as an explicit convolution
    w = np.ones(k) / k
    return np.convolve(y, w, mode="same")

# later, the library equivalent: scipy.signal.butter + filtfilt

The same smoothing operation, first as raw arithmetic and later as a calibrated library call. Understanding the first is what makes the second safe to use.

If you learn to read any sensor problem as measure, clean, represent, infer, fuse, deploy, you can walk into an unfamiliar modality and know where to start. That transfer is the point of the book.