Skip to main content
Back

Mercedes-Benz's Forecasting the Past Catches Self-Driving Blind Spots With 96% Accuracy

A post-hoc gradient trick spots when a trajectory predictor is about to fail, boosting AUROC from 56.8% to 71.0% on the Shifts benchmark.

  • New paper proposes gradient-based distribution shift detection for trajectory prediction without modifying the deployed model. arXiv
  • A frozen encoder feeds a small decoder trained to forecast the second half of the past trajectory from the first half.
  • The L2 norm of the forecasting loss gradient at the decoder's last layer serves as the anomaly score.
  • Achieves 71.0% AUROC on Shifts vs 56.8% baseline, and 70 to 79% AUROC on Argoverse OOD splits.
  • Detects Highway simulator collisions one second early with up to 96.1% AUROC on intersection scenarios.
  • Open source MIT-licensed implementation available on GitHub.

Trajectory prediction is the quiet workhorse of autonomous driving: every plan the car makes depends on guessing where the other vehicles, cyclists, and pedestrians will be a few seconds from now. The problem is that these predictors are trained on finite data, and the road throws curveballs. New city, new weather, a driver behaving strangely, and the model happily produces a confident but wrong forecast. A new paper from researchers at Friedrich-Alexander-Universität Erlangen-Nürnberg and Mercedes-Benz proposes a lightweight way to catch those moments before they become collisions.

The method, called Forecasting the Past, leaves the original prediction model alone. It attaches a small auxiliary decoder that learns a self-supervised task, then reads out the gradient of that decoder's loss as an anomaly score. On the Shifts vehicle motion dataset it lifts AUROC from around 56.8% to 71.0%, and on the Highway simulator it detects impending collisions with up to 96.1% AUROC on intersections.

Why trajectory predictors quietly break

Existing approaches to catching out-of-distribution (OOD) inputs in trajectory prediction fall into three camps, each with a known weakness. Reconstruction-based methods flag samples that produce high reconstruction error, one-class methods learn a boundary around normal data, and uncertainty quantification approaches estimate confidence with probabilistic estimation. In practice reconstruction error confuses input complexity with real anomalies, one-class methods need painful hyperparameter tuning on high-dimensional trajectory data, and uncertainty methods depend heavily on how well the model is calibrated.