Skip to main content
Back

WPI Strips Robot Arm Control Down to a Single Camera Using Inpainting

Researchers used image inpainting to erase training markers and reconstruct occluded robot parts, enabling markerless visual control without camera calibration or kinematic models.

Pro

  • Inpainting removes ArUco markers from training images, producing automatically labeled markerless keypoint data
  • Attention U-Net GAN with WGAN-GP reconstructs occluded robot parts at runtime without explicit masks
  • Unscented Kalman Filter smooths keypoint trajectories and predicts positions during detection failures
  • Achieves 100% keypoint detection within 10 pixels under full visibility on a Franka Panda arm
  • First eye-to-hand IBVS demo of 3D control using monocular RGB, no depth or robot model needed
  • Code and datasets released on GitHub and Zenodo

Controlling a robot arm by watching it move sounds simple, but most vision-based control pipelines lean heavily on fiducial markers, precise camera calibration, and accurate kinematic models. A team from Worcester Polytechnic Institute has shown you can strip all of that away by using image inpainting in two clever places: once to erase training markers from labeled data, and again at runtime to reconstruct parts of the robot hidden by obstacles.

The paper presents a visual servoing framework that controls a Franka Emika Panda arm using only natural keypoints on the robot's body, detected from a single fixed RGB camera. The authors claim this is the first eye-to-hand image-based visual servoing (IBVS) pipeline to achieve 3D motion control from one camera and natural 2D keypoints, without depth sensing or explicit robot or camera models.

Why markers are a bottleneck

Traditional IBVS depends on ArUco tags, AprilTags, or QR codes stuck to the robot. These fail under self-occlusion, motion blur, reflections, or bad lighting, and attaching them is impractical in many real-world applications. Borrowing techniques from human pose estimation runs into a data problem: there is no COCO-scale dataset of annotated robot keypoints.

Earlier attempts either trained in simulation (creating a sim-to-real gap) or leaned on camera intrinsics plus forward kinematics to project 3D joint frames into image space. That works for rigid arms with accurate models, but breaks for soft, underactuated, or cheaply built robots where the kinematic chain can't be trusted.