Skip to main content
Back

NYU Abu Dhabi Builds a Safety Layer That Stops Humanoids From Hitting You

A vision-based pipeline uses control barrier functions to filter human motion imitation commands in real time, blocking self-collisions and human-robot collisions before they happen.

Pro

  • NYU Abu Dhabi team proposes a vision-based safety framework for humanoid motion imitation from a single camera.
  • Control Barrier Function layer solves a QP each step to filter unsafe joint commands in real time.
  • Both robot and human are modeled as seven capsules; closest-point distance defines the safety barrier.
  • Handles self-collision and human-robot collision in one unified constraint set.
  • Capsule geometry on GPU with JAX gave the best solve time versus box and sphere decompositions.
  • Simulation only so far, upper body only, no code release, real-hardware transfer is future work.

Teaching a humanoid to mirror a person's movements is easy to demo and hard to make safe. Arms swing toward the torso, hands cross the chest, and the demonstrator often stands close enough to get clipped. Researchers from NYU Abu Dhabi have proposed a framework that keeps the imitation reactive while wrapping it in a mathematically enforced safety layer, using nothing more than a single camera for perception. The full write-up is available on arXiv.

Why imitation keeps hitting a wall

Most humanoid teleoperation setups still lean on motion capture suits or wearable IMUs. Those sensors are accurate but expensive, cumbersome, and impractical for lightweight daily deployment. Camera-based pose estimation is the accessible alternative, and it introduces a new problem: nothing in the perception-to-retargeting pipeline knows when the robot is about to smack itself or the person standing in front of it.

During real-time upper-body tracking, the robot may encounter both self-collision and human-robot collision, especially when arms move close to the torso or reach toward the demonstrator. Fixing this at the trajectory-planning level is too slow for streaming imitation. It has to happen inline, at control rate.

From pixels to safe joint commands