Section 14.3: Dilations and receptive fields

"You asked me to hear the arrhythmia and the arrhythmic minute it hides in. Same ears, two time scales. So I learned to skip."
A Multi-Scale AI Agent

Why this section matters

The temporal convolutional network of Section 14.2 gave us a causal, parallelizable alternative to recurrence, but it left one problem unsolved: a plain stack of convolutions reaches back into the past only linearly with depth, so seeing several seconds of a 100 Hz stream would demand a tower dozens of layers tall. Dilation is the trick that breaks that ceiling. By spacing a kernel's taps apart and doubling the spacing each layer, a modest stack grows its receptive field (the span of past samples one output can actually see) exponentially rather than linearly, which is what lets a compact network watch a single heartbeat and the rhythm it belongs to at the same time. Getting the receptive field to match the physics of your task, no shorter and not wastefully longer, is most of the practical craft in this chapter: compute it exactly, tune it deliberately, budget it against latency and memory.

This section builds on the causal, weight-shared convolution and single-layer receptive-field formula from Chapter 13, and on the residual TCN block of Section 14.2; converting samples to seconds uses the sampling relationships from Chapter 3. Notation: a window is $x \in \mathbb{R}^{C \times T}$ with $C$ channels and $T$ time steps, kernel size is $K$, and the dilation of a layer is $d$.

What dilation does to a kernel

A dilated convolution is an ordinary convolution whose kernel taps are spread out by inserting $d-1$ gaps between them. For a causal layer with dilation $d$, one output channel computes

$$z_o[t] = b_o + \sum_{c=1}^{C} \sum_{k=0}^{K-1} w_{o,c}[k]\, x_c\big[t - d\,(K-1-k)\big].$$

When $d=1$ this is the familiar dense causal convolution; each output reads $K$ consecutive past samples. When $d=2$, the kernel skips every other sample, so a size-3 kernel spans 5 input samples while still touching only 3 of them: a wider span at fixed parameter cost. The span, not the tap count, sets how far back the layer can look, and parameter count and multiply-accumulate cost depend only on $K$, never on $d$, so dilation buys reach for free in parameters and nearly free in compute. In every framework it is one argument: PyTorch's nn.Conv1d takes dilation=d, and you keep the layer causal by left-padding with $d\,(K-1)$ zeros so $z[t]$ never reaches a future sample, the online constraint a streaming model must honor (Section 14.5).

Dilation is not stride: the sequence never shrinks

It is easy to picture "skip samples" and conclude that dilation downsamples the sequence the way a strided convolution or pooling layer does. It does not. Stride subsamples the output, so $T$ shrinks at every such layer. Dilation subsamples only which input positions one kernel tap reads: every output still has one value per input time step, $T$ is unchanged end to end, and each sample stays individually addressable. Confusing the two produces real bugs, from feeding a downstream head data it expects to already be shorter, to assuming an alignment that only dilation guarantees.

Double the dilation, double the reach, keep the cost

Stack $L$ causal layers with the same kernel $K$ and geometrically increasing dilations $d_\ell = 2^{\ell-1}$ (that is, 1, 2, 4, 8, ...). The receptive field is

$$R = 1 + (K-1)\sum_{\ell=1}^{L} d_\ell = 1 + (K-1)\big(2^{L}-1\big).$$

Receptive field grows exponentially with depth; parameters and FLOPs grow only linearly with it. Each extra layer doubles how far back the network can see for the cost of one more layer, which is why TCNs reach multi-second context in a handful of layers. A plain stack ($d_\ell=1$) gives $R = 1 + L(K-1)$, linear in depth: with $K=3$, eight dilated layers see $1 + 2(2^{8}-1) = 511$ samples, eight dense layers only 17.

Budgeting the receptive field to the physics

The exponential formula is a design tool, not just an observation. Work backward from the task: decide the longest phenomenon a single prediction must integrate (cardiac rhythm, eight to ten beats, roughly 8 seconds; a gait transition, about 2 seconds; a bearing fault signature, several shaft revolutions), convert that duration to samples to get a target $R^{\star}$, then choose $K$ and $L$ so $R \ge R^{\star}$. Because $R$ is dominated by the largest dilation $2^{L-1}$, each added layer roughly doubles the horizon, so the layers you need scale with the logarithm of the required history. A subtle failure mode: with $K=2$ and aggressive dilation, consecutive taps at a deep layer sit $2^{L-1}$ samples apart, and information between them can fall through the cracks (gridding). The defenses are standard: use $K=3$ or larger so taps overlap between layers, repeat the dilation cycle across several stacked blocks instead of dilating once to a huge value, and cap the maximum dilation near the sample spacing of the feature you care about. Dilation is a coverage argument, not only a reach argument.

An ambulatory ECG detector that must span ten beats

A clinical wearables team is building an atrial-fibrillation screener from single-lead ECG at 250 Hz (the cardiac modeling this feeds into is Chapter 29). AF is a rhythm irregularity, not a single-beat morphology, so one decision must see roughly ten consecutive R-peaks: about 8 seconds, $R^{\star}\approx 2000$ samples, which a dense stack would need hundreds of layers to reach. Instead they use residual TCN blocks with $K=7$ and dilations 1, 2, 4, 8, 16, 32, 64, giving $R = 1 + 6\,(2^{7}-1) = 763$ samples per block; two stacked cycles push the field past 2000 samples, comfortably covering ten beats, in fourteen convolutional layers. The model runs causally so it can later stream on the patient's monitor, and a computed rather than guessed receptive field lets the team show regulatory reviewers (Chapter 34) exactly how much history each alarm depends on.

The helper below computes the receptive field of a dilation schedule and inverts the question a practitioner asks: given a sample rate and required history, how deep must the stack be? It makes this section's arithmetic executable, and the printed numbers match the ECG example above.

import math

def receptive_field(kernel, dilations):
    """Samples one output can see for a causal dilated stack."""
    return 1 + (kernel - 1) * sum(dilations)

def layers_needed(kernel, target_samples, blocks=1):
    """Smallest L with dilations 1,2,4,...,2^(L-1), repeated `blocks` times,
    whose receptive field covers target_samples."""
    L = 1
    while True:
        dils = [2 ** i for i in range(L)] * blocks
        if receptive_field(kernel, dils) >= target_samples:
            return L, receptive_field(kernel, dils)
        L += 1

fs = 250                       # ECG sample rate, Hz
history_s = 8.0                # need ~10 beats
target = int(fs * history_s)   # 2000 samples
L, R = layers_needed(kernel=7, target_samples=target, blocks=2)
print(f"layers per block: {L}, receptive field: {R} samples "
      f"= {R / fs:.1f} s")     # layers per block: 7, receptive field: 2033 samples = 8.1 s

Sizing a dilated TCN from physics: layers_needed inverts the receptive-field formula so you pick depth from a required history rather than by trial and error, confirming that seven dilated layers (kernel 7) over two blocks span 8.1 s of 250 Hz ECG.

Reach is not the whole story: cost, latency, and effective field

A large receptive field is necessary but not sufficient. Three costs travel with it. Latency and buffering: a causal model with receptive field $R$ needs $R-1$ past samples buffered to produce the current output, so streaming memory and warm-up time before the first valid prediction both scale with $R$ (made explicit in Section 14.5). Compute: dilation leaves per-layer FLOPs unchanged, but every layer still runs at every time step, so doubling the horizon by adding a layer adds one layer's work across the whole sequence. Effective versus theoretical field: the box formula gives the span a layer can reach, not the span it does use. A far-back sample's influence on the output is a product of many learned weights, one per layer it must cross, and trained weights typically have magnitude well under 1, so that influence decays roughly geometrically with distance. The result, measurable by zeroing one past input at a time and watching the output change, is an effective field smaller than the theoretical one and bell-shaped around the recent past. This bites hardest where the theoretical field is sized with no slack, as in the AF window above, which is why provisioning two to three times your target buys real headroom rather than an exact match that training quietly erodes. When even a generous dilated stack cannot hold the horizon you need, reach for the recurrent state of Section 14.1, the global attention of Chapter 15, or the linear-time models of Chapter 16.

Learning the dilation schedule instead of hand-setting it

The geometric schedule $d_\ell = 2^{\ell-1}$ is a good default, not a law. An active line of work treats per-layer dilation as something to search or learn: architecture search over candidate schedules, and differentiable relaxations that let a layer interpolate between dilation rates during training and settle on whichever the gradient prefers. The appeal for sensor time series is that the right schedule is physics-dependent (a bearing fault and a respiration cycle want very different tap spacing), so a per-channel, learned schedule could remove a manual tuning step without giving up TCN parallelism.

Dilation is one keyword, not a loop

Implementing a dilated causal convolution by hand means building the gapped index pattern, computing the left pad $d(K-1)$, slicing off the extra samples the pad introduces, and threading it through autograd: roughly 20 to 30 lines, easy to get subtly wrong on the causal boundary. In PyTorch it is nn.Conv1d(c_in, c_out, kernel_size=K, dilation=d, padding=d*(K-1)) plus trimming the final $d(K-1)$ outputs; the framework handles strided memory access, gradients, and GPU kernels. One keyword, dilation=d, replaces the entire hand-rolled index scheme.

Exercise: design a stack for a vibration monitor

You are monitoring a motor with an accelerometer at 2 kHz. The bearing fault signature repeats once per shaft revolution, and the shaft turns at 1800 rpm, so each prediction should integrate at least five revolutions. (1) Convert five revolutions to a target receptive field in samples. (2) With kernel $K=3$ and geometric dilations starting at 1, how many layers does one dilation cycle need, using $R = 1 + (K-1)(2^{L}-1)$? (3) Recompute with $K=5$ and comment on the parameter-versus-depth tradeoff. (4) The target is large enough that a single cycle reaches a dilation of hundreds of samples between adjacent taps. Explain the gridding risk and how repeating the cycle across two blocks mitigates it. Show every number.

Self-check

Two stacks use kernel $K=3$: stack A has ten dense layers ($d=1$); stack B has ten layers with $d=2^{\ell-1}$. Compute each receptive field, state the ratio, then explain in one sentence why B costs no more parameters than A.
A colleague sets $K=2$ and dilations 1, 4, 16, 64 to save compute, and finds the model misses events at odd sample offsets. Name the failure and give two changes that fix it without shrinking the receptive field much.
Your theoretical receptive field is 4 seconds but validation shows the model ignores anything older than about 1.5 seconds. Name this phenomenon, and say what it implies for how much theoretical field to provision relative to the history you truly need.

What's Next

In Section 14.4, we turn from how far a model can see to what it emits: the distinction between collapsing a whole window into a single label and producing an aligned output at every time step. The receptive-field budgeting you just learned decides which of those framings a given dilated stack can support, and how much context each per-step prediction is allowed to draw on.