Synthetic data lets you generate labeled sensor streams from a model of the world instead of collecting them in the field. For perception systems this is attractive for three reasons: rare events that you cannot wait to observe (a pedestrian stepping off a curb at night, a bearing seconds before it seizes), labels that are expensive or impossible to annotate by hand (dense depth, per-point semantics, exact 6-DoF pose), and privacy, where a rendered face or a simulated home avoids consent and retention problems entirely. This appendix is a working reference to the tools and the discipline that make synthetic sensor data actually usable.
Physics-based simulators and noise models
A synthetic sensor sample is only as good as its sensor model. The clean output of a renderer or a ray tracer is the signal; the sensor model is what turns it into a measurement. For a camera this means adding shot noise, read noise, fixed-pattern noise, rolling-shutter skew, motion blur, lens distortion, and a response curve. A common image-formation model is
$$ I = f\!\left( g \cdot (\Phi + N_{\text{shot}}) + N_{\text{read}} \right), $$where \(\Phi\) is the photon count, \(g\) is analog gain, \(N_{\text{shot}} \sim \mathrm{Poisson}\) scales with \(\Phi\), \(N_{\text{read}}\) is Gaussian, and \(f\) is the tone curve plus quantization. Lidar needs beam divergence, range-dependent dropout, intensity falloff, and multi-echo returns on edges; radar needs multipath and speckle; an IMU needs bias random walk and scale error. Getting these right matters more than photorealism: a detector trained on noise-free renders learns to depend on a cleanliness that no real sensor delivers.
Digital twins
A digital twin is a simulation kept in correspondence with a specific physical system: this exact factory line, this vehicle, this wind turbine, driven by its real geometry, materials, and telemetry. Twins are the natural host for synthetic data because they let you replay recorded operating conditions and then perturb them, generating counterfactual sensor streams for states the real asset has not yet reached. They are the standard vehicle for predictive-maintenance data, where you need failure signatures that, by definition, you rarely get to record.
Domain randomization and sim-to-real
The core sim-to-real problem is covariate shift: the synthetic distribution \(p_{\text{sim}}(x)\) differs from the real one \(p_{\text{real}}(x)\), so a model tuned to the former transfers poorly. Domain randomization attacks this by deliberately over-varying the nuisance factors (textures, lighting, camera pose, object placement, noise levels) so that the real world looks like just another sample of the training distribution. The complementary route is domain adaptation: align features or translate images so synthetic and real embeddings overlap. In practice you combine them, randomize what you cannot measure and calibrate what you can.
Simulators and engines
| Tool | Primary domain | Sensors | Note |
|---|---|---|---|
| NVIDIA Isaac Sim / Isaac Lab | Robotics, manipulation | RGB-D, lidar, IMU, contact | PhysX + RTX rendering; large-scale RL and synthetic-data workflows |
| CARLA | Autonomous driving | Camera, lidar, radar, GNSS | Open-source urban driving; scripted traffic and weather |
| Gazebo | General robotics | Camera, lidar, sonar, IMU | ROS-native; fast, lower-fidelity rendering |
| MuJoCo | Contact dynamics, control | Proprioception, touch, RGB | Accurate physics for legged and dexterous control |
Choose by what you need to be faithful. If contact forces drive your task, MuJoCo's solver matters more than its pixels; if you need photoreal traffic scenes, CARLA or Isaac Sim earn their heavier render cost.
Neural-field and Gaussian-splatting re-simulation
A newer family builds the scene from real captures rather than hand-authored assets. Neural radiance fields (NeRF) and 3D Gaussian splatting reconstruct a scene from a set of images or lidar sweeps, then let you re-render it from new viewpoints, new times, or with edited actors. This closes much of the appearance gap because the textures, lighting, and geometry are the real ones. Lidar-specific variants (for example neural fields fit to point clouds) re-simulate range returns for novel trajectories, which is how you generate the "same intersection, different approach angle" data that pure asset libraries cannot.
Controllable fault injection
Synthetic pipelines let you script failure. Inject stuck pixels, dropped lidar returns, IMU bias jumps, timestamp jitter, partial occlusion, or a cracked lens, each with a known label and severity. This is where synthetic data has no real rival: you can produce a balanced, exhaustively labeled catalog of fault modes for a robustness or anomaly-detection benchmark that would take years to collect in the wild.
def inject_lidar_faults(points, dropout=0.05, range_bias=0.0, rng=None):
rng = rng or np.random.default_rng()
keep = rng.random(len(points)) > dropout # random return loss
pts = points[keep].copy()
pts[:, :3] *= (1.0 + range_bias) # systematic range error
return pts
A minimal, seed-controlled lidar fault injector: reproducible dropout plus a systematic range bias, each recorded so the perturbation itself becomes a label.
Validation: does the synthetic set transfer?
The failure mode of synthetic data is silent: a set that looks convincing but does not transfer, so a model scores well in sim and fails on the road. Never ship a synthetic pipeline without a transfer check. Useful protocols:
- Train-sim, test-real (and the reverse). The honest headline number. A large gap between sim validation and real test accuracy signals unmodeled shift.
- Real-vs-synthetic discrimination. Train a classifier to tell real from synthetic. If it wins easily, some feature is a giveaway; if it approaches chance, the marginals are matched.
- Statistical distance on features. Compare embeddings with Frechet distance or MMD rather than raw pixels.
- Real-data augmentation curves. Plot real-test accuracy as you add synthetic samples. A curve that plateaus early or bends down means the synthetic set is adding bias, not information.
When it helps, and when it misleads
Synthetic data earns its place for rare events, dense labels, controlled faults, and privacy-sensitive settings, and as a pretraining or augmentation source that a smaller real set then corrects. Its failure modes are consistent: an unrealistic sensor-noise model, textures and materials that are too clean, a physics engine that does not match the real contact or optics, and a long tail of real-world conditions the simulation never anticipates. Treat every synthetic corpus as a hypothesis about the real distribution, and keep a held-out real test set as the referee. See Appendix F for the physical sensor specifications these models approximate, and Chapter references on sim-to-real for the training recipes that pair with these tools.