Skip to main content
Back

SynOE-OD Teaches Self-Driving Cars to Spot Objects They've Never Seen

A new framework teaches street-scene detectors to spot unknown objects by pasting AI-generated oddities into training images, beating GroundingDINO on the OoDIS benchmark.

Pro

  • SynOE-OD teaches a single detector to jointly find known and unknown objects in street scenes.
  • Uses Stable Diffusion to inpaint unusual objects and GroundingDINO to auto-label them.
  • ChatGPT supplies the list of "unusual for a street" categories: animals, household items, robots.
  • Beats zero-shot GroundingDINO on the OoDIS benchmark by 5 to 10 AP points.
  • Best config: DINO-DETR with Swin-L fine-tuned on inpainted NuImages plus Cityscapes.
  • Trade-off: fine-tuned GroundingDINO loses about 7 mAP on ID classes due to noisy labels.

Self-driving cars have a quiet failure mode that rarely shows up in benchmark tables: when an object detector encounters something it has never seen, it often does not misclassify it. It just does not see it at all. A stray sofa on the highway, a hamster in the crosswalk, an overturned crate on the shoulder, all get silently absorbed into the background. A team from the University of Wuppertal, University of Osnabrück, and Aptiv has proposed SynOE-OD, a framework that fixes this by teaching a single detector to flag those unknowns while still doing its regular job.

The paper names the core problem plainly: atypical objects vanish into the background class, and existing out-of-distribution approaches lean on complex architectures or auxiliary branches that never unify in-distribution and OOD handling. Prior work in image classification popularized Outlier Exposure, where you show the model examples of things it should treat as unknown during training. Extending that to object detection has been awkward, mostly because there is no easy source of realistic, labeled OOD objects sitting inside street scenes.

Painting weirdness into the training set

SynOE-OD sidesteps the data problem by manufacturing it. The pipeline uses Stable Diffusion to inpaint unusual objects into real driving images, then uses GroundingDINO, an open-vocabulary detector, to relabel the results. The list of what counts as unusual comes from asking ChatGPT for objects that would look out of place on a street, which returns categories dominated by household items and animals: penguins, sofas, hamsters, printers, and so on.