Skip to main content
Back

Tongji's SPR Teaches Self-Driving AI to See Real Roads From Fake Ones

A new domain-adaptation method for driving segmentation preserves semantic structure between synthetic and real images, boosting mIoU by up to 20.7 points.

Pro

Paper, Full Access
  • SPR adds inter-class and intra-class correlation losses on top of prototype contrastive learning for driving segmentation
  • Reaches 54.4% mIoU on GTA5 to Cityscapes, a 17.3-point gain over source-only baseline
  • Scores 49.5% mIoU on Cityscapes to ACDC adverse-weather task, beating Refign and VBLC
  • Excels on safety-critical classes: 97.9% road, 91.5% building, 84.7% sky under fog and night
  • Adds zero inference overhead; runs at 23 FPS with ResNet-101, 43.4 ms latency
  • Weak on long-tail classes like train and traffic sign; source-free variant lags leading SFUDA methods

Self-driving perception systems face a stubborn economics problem. Labeling a single street scene in the Cityscapes benchmark takes roughly 1.5 hours under normal conditions and up to 3.3 hours in bad weather, per pixel. Synthetic datasets like GTA5 and SYNTHIA sidestep the annotation bill entirely, but models trained on video-game frames tend to fall apart the moment they see a real camera feed. A new paper from Tongji University and collaborators, Structured Prototype Regularization (SPR), proposes a way to close that gap by teaching the network to preserve the shape of its own feature space when moving between domains.

Where existing sim-to-real adapters go wrong

Unsupervised domain adaptation (UDA) for segmentation usually works by aligning the overall distribution of features from synthetic images with the distribution from real ones, often via an adversarial discriminator. The authors argue that aligning global distributions alone does not guarantee effective class separability in the shared feature space. In practice, a road pixel and a sidewalk pixel can end up neighbors in feature space even after adaptation, which is exactly the kind of confusion that produces dangerous segmentation errors.

Prototype-based contrastive methods improved things by anchoring each class to a learned centroid, though they typically treat those centroids as isolated points and ignore the structural relationships between them. SPR's core claim is that the geometry between prototypes, how far apart classes sit and how tightly each class clusters, matters as much as their individual positions.