Formats for telemetry, waveforms, event streams, and point clouds

"They handed me a folder of float32 arrays with no sample rate, no units, and no timezone. I did not have a dataset. I had a rumor."

A Disillusioned AI Agent

Four shapes, not one

Pick the wrong shape and the damage is silent: a waveform forced into CSV bloats tenfold and a resampled event stream loses the very timing that was its signal, defects nobody catches until a model trained on the corrupted data misbehaves in the field. Almost every sensor stream you will meet falls into one of four structural families, distinguished by three questions: are the samples uniformly spaced in time, is the data dense or sparse, and does each record hold a fixed or variable number of values? Get the family right and the storage decision nearly makes itself.

Telemetry is low-rate, multivariate, timestamped scalars: engine revolutions per minute (RPM), battery voltage, Global Positioning System (GPS) latitude, cabin temperature. Each is a named channel sampled from a fraction of a hertz up to roughly a hundred. It is naturally a wide table, one row per timestamp, one column per channel. Waveforms are dense, uniformly sampled continuous signals: audio at 44.1 kHz, an electrocardiogram (ECG) trace at 500 Hz, a bearing accelerometer at 25.6 kHz. Here the sample rate is not data to store per row; it is a single number that, together with a start time, lets you reconstruct every timestamp implicitly. Event streams are asynchronous: things happen at irregular instants and you record a tuple when they do. A CAN message, a syslog line, a keystroke, or an event-camera pixel firing \((x, y, \text{polarity}, t)\) (polarity being whether that pixel just brightened or darkened) all share the trait that time between records is itself information. Point clouds are unordered sets. A single lidar sweep is a variable-size bag of points, each carrying \((x, y, z)\) plus intensity and often a per-point timestamp. The points have no meaningful row order, and every frame holds a different count.

Table 5.1.1 lays these four side by side. The reason this taxonomy matters is that each shape breaks a different default. Telemetry punishes row-major storage when you only need two of two hundred channels. Waveforms punish any format that stores a timestamp per sample. Event streams punish fixed-rate assumptions. Point clouds punish anything that expects a rectangular array. Choosing the layout is choosing which of these mistakes you refuse to make. In short: the shape of a sensor stream, not your convenience, dictates the format that keeps its physics alive. Figure 5.1.1 illustrates the four canonical sensor-data shapes and their fitting storage layouts.

Rows, columns, and chunks: why layout is destiny

The deepest split in sensor storage is row-major versus columnar. A row-major file (CSV, JavaScript Object Notation (JSON) lines, a naive database table) stores all fields of record 1, then all fields of record 2, and so on. A columnar file (Parquet, Arrow, ORC) stores all values of channel A contiguously, then all of channel B. Figure 5.1.1 draws the two layouts side by side and shows what happens when a query asks for a single channel. For telemetry the columnar layout is transformative for three concrete reasons. First, selective reads: an analyst who wants only rpm and oil_temp from a two-hundred-channel truck log reads two columns off disk and skips the rest, instead of parsing every byte of every row (the right half of Figure 5.1.1). Second, compression: values within one channel are similar (temperature changes slowly), so a column of them compresses far better than interleaved heterogeneous rows; run-length and dictionary encoding (schemes that replace long runs of a repeated value, and repeated values themselves, with short codes) routinely shrink telemetry by five to twenty times. Third, predicate pushdown: column statistics stored per block let a reader skip entire chunks whose value range cannot match a filter, so "give me the rows where RPM exceeded 6000" touches only the blocks that could contain them.

Parquet, named repeatedly above, is the concrete workhorse worth pinning down: it is an open columnar file format that stores each channel as its own compressed, statistics-tagged block inside a single self-contained file. It matters because it delivers all three columnar wins (selective reads, per-column compression, predicate pushdown) with no custom code, which is why it has become the default container for tabular telemetry. Mechanically, it partitions the table into "row groups" and, inside each, writes every column contiguously with a dictionary and min/max statistics, so a reader can seek straight to the exact columns and blocks a query needs. Reach for Parquet on wide, sparse, mixed-type streams like telemetry and event logs; prefer a chunked-array format such as HDF5 or Zarr when the data is instead a single dense uniform waveform.

Waveforms want a different discipline. A dense uniform signal carries no per-sample metadata worth storing, so the right structure is a chunked array: one contiguous block of a single dtype (data type, for example 32-bit float or 16-bit integer, describing how each sample's bytes are interpreted), split into fixed-size chunks along the time axis, with sample rate, units, dtype, start time, and device identity held once as attributes. HDF5 and Zarr do exactly this; WAV and the biosignal EDF (European Data Format) container do a domain-specific version. Chunking makes a multi-hour recording streamable: you read the ten-second window you need without materializing the whole file, the access pattern Section 5.2 builds windows on. The governing rule is that timestamps for a uniform signal are computed, never stored: sample \(n\) sits at \(t_0 + n/f_s\), so a stored timestamp column wastes space and lets rounding error desynchronize the axis. Getting \(t_0\) and \(f_s\) right ties back to the synchronization discipline of Chapter 3.

Event streams and point clouds: the irregular families

Telemetry and waveforms both eventually submit to a grid, whether a column table or an implied uniform time axis; the remaining two families never do. Event streams and point clouds resist the tidy rectangle, and pretending otherwise is where naive pipelines quietly corrupt data. An event stream carries its meaning in the gaps: a burst of CAN messages during hard braking and silence during cruise are both signal. Store events as a row log where the timestamp is a first-class column, never resample them onto a fixed grid at ingestion time (that decision belongs downstream and destroys information if baked in), and keep them monotonically ordered by time. Robotics leans on MCAP and the older rosbag containers precisely because they log heterogeneous, asynchronous messages with per-message timestamps and typed schemas; neuromorphic vision uses AEDAT and EVT to pack millions of \((x, y, p, t)\) tuples per second, the format that Chapter 46 builds event-camera perception on. A columnar log such as Parquet also serves general event data well, since the timestamp and payload columns compress and filter efficiently.

If an event stream scatters its records along time, a point cloud scatters them across space, and it defeats the rectangle for reasons of its own. A point cloud is a set, and sets have two properties that break array-shaped storage: variable cardinality (each sweep has a different point count) and permutation invariance (no canonical order). You cannot stack frames into one rectangular tensor without padding, and you must never let a tool assume row order carries meaning. Automotive lidar pipelines commonly store each frame as a packed little-endian binary blob of \(N \times 4\) float32 values \((x, y, z, \text{intensity})\) with \(N\) recorded in a sidecar index; geospatial workflows use LAS and its compressed cousin LAZ; general 3D tooling uses PCD and PLY. Whatever the container, the metadata that must ride along is the sensor pose, the coordinate frame, and the per-point time offset, because a sweep is not instantaneous and the vehicle moved while it scanned. The models that consume these sets, and the reason their architectures must be permutation-invariant, are the subject of Chapter 42.

Self-describing or worthless

Across all four shapes, one property separates a dataset from a rumor: the file must carry enough metadata to reconstruct physical meaning without external knowledge. The minimum viable descriptor is the tuple in the code below, and its absence is the single most common way sensor data arrives broken. You cannot interpret a bare float32 array without its dtype and endianness (is that little-endian float or big-endian int?), its sample rate and start time (or the time axis is unknowable), its units and scale (millivolts or volts, raw analog-to-digital converter (ADC) counts or calibrated), its device and session identity (or you cannot later build the device-disjoint splits that Section 5.4 requires), and its timezone or an unambiguous epoch (or midnight is a coin flip). Listing 5.1.1 writes a waveform to Parquet with exactly this descriptor attached as schema metadata, then reads it back and reconstructs the time axis from \(f_s\) and \(t_0\) alone.

import numpy as np, pyarrow as pa, pyarrow.parquet as pq

fs, t0 = 25600.0, 1_726_000_000.0          # Hz, and a UTC epoch second
sig = np.random.randn(256_000).astype(np.float32)   # ~10 s of vibration

# Attach the self-describing descriptor as column-schema metadata.
table = pa.table({"accel_axial": sig})
table = table.replace_schema_metadata({
    b"sample_rate_hz": b"25600.0",
    b"start_epoch_utc": b"1726000000.0",
    b"units": b"m/s^2",
    b"device_id": b"turbine-044-gearbox",
    b"session_id": b"2026-07-14T09:00Z",
})
pq.write_table(table, "vib.parquet", compression="zstd")

# A reader anywhere reconstructs the time axis with no side channel.
back = pq.read_table("vib.parquet")
meta = back.schema.metadata
fs_r = float(meta[b"sample_rate_hz"])
n    = back.num_rows
t    = float(meta[b"start_epoch_utc"]) + np.arange(n) / fs_r   # implicit, exact

As Listing 5.1.1 shows, carrying device_id and session_id is not bookkeeping pedantry. Those fields are the hooks that make leakage-safe splitting possible at all; drop them at ingestion and no downstream care can recover a device-disjoint test set, which is why Section 5.3 treats missing provenance as a first-class dataset defect.

import pandas as pd
df.to_parquet("telemetry.parquet", compression="zstd")   # schema, stats,
                                                          # chunking, all handled
cols = pd.read_parquet("telemetry.parquet", columns=["rpm", "oil_temp"])

What's Next

Section 5.2 takes these correctly-stored streams and carves them into the windows a model actually trains on: how long a window should be, how much they overlap, and how a label gets attached to a slice of time without smearing across boundaries. The self-describing descriptors you built here are exactly what make that windowing reproducible instead of a pile of magic numbers.