Rutgers Maps 435 Papers Showing Where Foundation Models in Robotics Are Stuck
A 435-paper survey maps how foundation models are reshaping robotics through five distinct research phases and a six-criteria taxonomy.
Takeaways
- New survey analyzes 435 papers on foundation models in robotics across six taxonomic criteria and five research phases.
- Field has evolved from bolt-on NLP/CV modules to end-to-end Vision-Language-Action generalist policies.
- Compares LLMs, VFMs, VLMs, VLAs plus transformer, Mamba, diffusion, CNN, and graph architectures.
- Public datasets like Open X-Embodiment, DROID, and AgiBot World now scale to millions of trajectories.
- Key blockers: 50Hz control loops vs seconds of inference, no standard action space across embodiments.
- Tactile data, failure/recovery examples, and unified evaluation benchmarks remain critically missing.
Robotics is going through the same kind of shift that language models triggered in NLP a few years ago. A new review from Harokopio University, Rutgers, IIT Genova and collaborators tries to make sense of that shift by mapping the research landscape of foundation models in robotics across 435 papers, and it lands on a clean story about where the field has been, where it is stuck, and what comes next.
The authors argue that robotics is transitioning from fixed, single-task, domain-specific solutions toward adaptive, general-purpose agents capable of operating in complex, open-world environments, with foundation models (FMs) as the main driver. Their contribution is a structured way to compare the systems that already exist rather than a new model of their own.
Five phases that got us to generalist robots
The review organizes the last several years into five distinct research phases. The trajectory runs from bolting off-the-shelf NLP and vision models onto classical controllers, through grounded vision-language planning, into end-to-end Vision-Language-Action (VLA) policies, then memory and web-to-robot transfer, and finally to multi-sensory generalist systems being deployed in the real world.
- Phase 1 (2018-2021): Native NLP and CV models play a supporting role while robot behavior remains determined by a hand-engineered downstream pipeline.
- Phase 2 (2021-2022):