This appendix collects the metrics used throughout the book in one place, with the definitions and formulas you will reach for while reading a results table. A metric answers a specific question, and using the wrong one hides the failure you care about. Keep one distinction in mind everywhere below: a task metric scores how well the model maps inputs to outputs on held-out data, while an operational metric scores how the deployed system behaves in the field: latency, throughput, uptime, alert volume, false-alarm rate per shift, cost per inference. A detector with excellent mAP that fires ten false alarms an hour is a task success and an operational failure. Report both, and never let one stand in for the other.
Classification
All the binary metrics build on the confusion-matrix counts: true positives \(TP\), false positives \(FP\), true negatives \(TN\), false negatives \(FN\).
| Metric | Formula | Reads as |
|---|---|---|
| Accuracy | \(\dfrac{TP+TN}{TP+TN+FP+FN}\) | Fraction correct; misleading under class imbalance. |
| Precision | \(\dfrac{TP}{TP+FP}\) | Of the positives you flagged, how many were real. |
| Recall | \(\dfrac{TP}{TP+FN}\) | Of the real positives, how many you caught. |
| F1 | \(\dfrac{2\,PR}{P+R}\) | Harmonic mean of precision \(P\) and recall \(R\). |
| Balanced accuracy | \(\tfrac{1}{2}\big(\text{TPR}+\text{TNR}\big)\) | Mean per-class recall; fair under imbalance. |
ROC-AUC is the area under the true-positive-rate versus false-positive-rate curve as the decision threshold sweeps; it equals the probability that a random positive scores above a random negative, and 0.5 is chance. PR-AUC is the area under the precision-recall curve and is the more honest summary when positives are rare, because ROC-AUC can look strong while precision at any usable threshold is poor. Prefer PR-AUC for imbalanced sensing problems (defect detection, rare-event alerting).
Object Detection
Intersection over union scores how well a predicted box overlaps a ground-truth box; a prediction counts as correct when its IoU with a true box exceeds a threshold (commonly 0.5).
\[ \text{IoU} = \frac{\lvert B_{\text{pred}} \cap B_{\text{gt}} \rvert}{\lvert B_{\text{pred}} \cup B_{\text{gt}} \rvert} \]Average precision (AP) is the area under the precision-recall curve for one class at one IoU threshold; mean average precision (mAP) averages AP over all classes. The COCO convention reports mAP averaged over IoU thresholds from 0.50 to 0.95 in steps of 0.05, a stricter score than the single-threshold mAP@0.5 common in earlier benchmarks. Always state which convention a number uses.
Forecasting and Regression
For point forecasts \(\hat{y}_i\) against truth \(y_i\) over \(n\) points:
\[ \text{MAE} = \frac{1}{n}\sum_i \lvert y_i - \hat{y}_i \rvert, \qquad \text{RMSE} = \sqrt{\frac{1}{n}\sum_i (y_i - \hat{y}_i)^2} \]RMSE penalizes large errors more heavily, so it is the metric of choice when big misses are disproportionately costly; MAE is more robust to outliers. The percentage errors trade absolute units for scale-free comparison:
\[ \text{MAPE} = \frac{100}{n}\sum_i \frac{\lvert y_i - \hat{y}_i \rvert}{\lvert y_i \rvert}, \qquad \text{sMAPE} = \frac{100}{n}\sum_i \frac{\lvert y_i - \hat{y}_i \rvert}{(\lvert y_i \rvert + \lvert \hat{y}_i \rvert)/2} \]MAPE explodes when actuals approach zero and punishes over-forecasts more than under-forecasts; sMAPE bounds the denominator but is still asymmetric and undefined when both terms are zero. For probabilistic forecasts, the continuous ranked probability score (CRPS) generalizes MAE to a predicted distribution with cumulative function \(F\), rewarding forecasts that are both sharp and calibrated:
\[ \text{CRPS}(F, y) = \int_{-\infty}^{\infty} \big(F(t) - \mathbb{1}\{t \ge y\}\big)^2 \, dt \]Anomaly Detection
Anomaly scores are thresholded to raise alerts, so the operational reality is an alert budget: an operator can only act on so many alerts per shift. Evaluate precision and recall at a fixed budget (for example, precision@100 alerts per day) rather than at an abstract optimal threshold that no one would pick in production.
Beware the point-adjust pitfall. A widely copied protocol marks an entire ground-truth anomaly segment as detected if the model flags any single timestamp inside it. This inflates recall and F1 so severely that a random score can beat published methods. Report unadjusted event-level precision and recall, or a range-aware metric, and treat any headline number that relies on point-adjustment as unreliable.
Calibration and Uncertainty
A model is calibrated when its stated confidence matches its empirical accuracy. Expected calibration error bins predictions by confidence and averages the gap between confidence and accuracy across \(M\) bins:
\[ \text{ECE} = \sum_{m=1}^{M} \frac{\lvert B_m \rvert}{N} \, \big\lvert \operatorname{acc}(B_m) - \operatorname{conf}(B_m) \big\rvert \]The Brier score is the mean squared error of predicted probabilities \(p_i\) against binary outcomes, a proper scoring rule that rewards honest probabilities:
\[ \text{Brier} = \frac{1}{n}\sum_i (p_i - y_i)^2 \]Conformal prediction turns any model into one with distribution-free coverage guarantees. Report two numbers together: empirical coverage, the fraction of test targets landing inside the prediction interval (which should match the nominal level, say 90%), and average interval width. Coverage without width is meaningless, since an infinitely wide interval always covers; the goal is nominal coverage at the smallest width.
Multi-Object Tracking
Tracking metrics score a set of predicted trajectories against ground-truth tracks over time. MOTP measures localization quality: the average overlap error of matched detections. MOTA aggregates the three failure modes of association: misses, false positives, and identity switches, where \(g_t\) is the number of ground-truth objects at frame \(t\).
\[ \text{MOTA} = 1 - \frac{\sum_t (\text{FN}_t + \text{FP}_t + \text{IDSW}_t)}{\sum_t g_t} \]MOTA can go negative when errors exceed the object count. It underweights identity consistency, which is why HOTA and IDF1 are now reported alongside it when track identity matters.
Localization and SLAM
For a trajectory estimate against ground truth, absolute trajectory error (ATE) measures global consistency as the root-mean-square position difference after aligning the two trajectories:
\[ \text{ATE} = \sqrt{\frac{1}{n}\sum_i \lVert \mathbf{p}_i - \hat{\mathbf{p}}_i \rVert^2} \]Relative pose error (RPE) measures local drift over a fixed time or distance interval \(\Delta\), which is the right metric for odometry where global alignment hides accumulating error. ATE answers "how far off is the whole map"; RPE answers "how fast does the estimate drift".
Remaining Useful Life
RUL prognostics predict how many cycles remain before failure. RMSE is the symmetric baseline, but in maintenance the two error directions are not equally costly: a late prediction (estimating more life than remains) risks unplanned failure, while an early prediction wastes usable life. The NASA C-MAPSS scoring function encodes this asymmetry, penalizing late predictions more steeply than early ones, with error \(d_i = \hat{y}_i - y_i\):
\[ S = \sum_i \begin{cases} e^{-d_i/13} - 1 & d_i < 0 \ (\text{early}) \\[2pt] e^{\,d_i/10} - 1 & d_i \ge 0 \ (\text{late}) \end{cases} \]Report RMSE for interpretability and the scoring function for operational realism; a model can improve one while worsening the other.
Fairness and Operational Metrics
When a perception system affects people, evaluate it per subgroup, not just in aggregate. Common criteria compare error rates across groups: demographic parity asks for equal positive rates, equalized odds asks for equal true-positive and false-positive rates across groups, and the gap between the worst and best subgroup on your primary metric is a blunt but useful summary. A face or gait system with strong overall accuracy can still fail badly for an underrepresented group, and only disaggregated reporting reveals it.
Operational and field-reliability metrics close the loop between a benchmark and a deployment.
| Metric | What it captures |
|---|---|
| Latency (p50, p95, p99) | Response time; tail percentiles matter more than the mean. |
| Throughput | Inferences or frames processed per second at steady state. |
| False alarms per shift | Operator-facing nuisance rate; drives whether alerts get trusted. |
| MTBF / MTTR | Mean time between failures and mean time to recover. |
| Availability | Uptime fraction; \(\text{MTBF}/(\text{MTBF}+\text{MTTR})\). |
| Cost per inference | Compute, energy, and bandwidth per decision. |
The recurring lesson: a model that wins on the task metric can still lose in the field. Choose the metric that matches the decision the system exists to support, report task and operational numbers side by side, and state the exact protocol (threshold, IoU convention, alert budget, adjustment) behind every number so a reader can trust the comparison.