Skip to main content
Back

University of Sydney Says Modular Self-Driving Is Dead, Neural Nets Win

A survey paper traces how autonomous driving stacks have collapsed from modular perception-planning-control pipelines into single neural networks trained on fleet-scale data.

Pro

Paper, Full Access
  • Survey argues autonomous driving has shifted from modular pipelines to single neural networks mapping pixels directly to controls.
  • Tesla FSD V14 is roughly 30x larger than V12, adds audio input, and reportedly uses a Mixture of Models router.
  • Standard recipe: imitation learning on fleet data, then RL on edge cases and simulated rare events.
  • NVIDIA released Alpamayo, an open-source ~10B parameter E2E driving agent plus 1700 hours of driving data.
  • Tesla claims 5.1M miles per major collision on FSD Supervised, but fair baselines shrink the safety gain to about 1.7x.
  • Same architecture is transferring to humanoids via Tesla Optimus and NVIDIA Project GR00T.

For twenty years, self-driving cars looked like a plumbing project. Engineers stitched together perception, prediction, planning and control as separate modules, each with its own interfaces, hand-tuned heuristics and calibration schedules. A new survey from the University of Sydney argues that architecture is now being retired in production, replaced by monolithic neural networks that map camera pixels directly to steering and throttle commands. The paper tracks how Tesla, Rivian, NVIDIA and Waymo are converging on end-to-end (E2E) learning, and what that shift means for the emerging product category the authors call FSD Supervised or L2++.

Why the old stack ran out of road

The classical pipeline is easy to draw and hard to scale. Sensing feeds mapping and localization, which feeds perception and prediction, which feeds a planner that hands trajectories to a controller. Each stage is debuggable in isolation, but the interfaces (bounding boxes, tracks, lane graphs) throw away information the downstream module might need.

The survey lists three structural problems that pushed the industry away from this design:

  • The long tail: the space of driving scenarios is effectively unbounded, so hand-coding rules for every edge case is impractical.
  • Sensor and infrastructure cost from multi-beam LiDAR, radar and continuously updated HD maps.
  • Ongoing calibration and maintenance overhead that is tolerable for a robotaxi fleet but not for consumer vehicles.
Diagram comparing modular pipeline with end-to-end architecture where gradients flow across all stages

What a Large Driving Model actually is

The authors frame the new approach as Large Driving Models (LDMs), an analogy to LLMs. A single network takes raw camera images, kinematic state, maps, a desired trajectory and in some cases radar, laser or audio, and directly outputs steering angles and acceleration commands, with gradients flowing from the driving loss all the way back to raw sensor inputs.

The paper is explicit about why this now works when it didn't in 2004 or 2016. The renaissance is scale: trained on millions of hours of fleet data, E2E models show emergent behaviors that were never programmed, including responses to unusual pedestrians, construction zones, emergency vehicles and adverse weather.