Skip to main content
Back

Harvard and MIT's Robot Learns When to Trust Touch Over Vision

A new diffusion policy trains separate experts per sensor and lets a router network decide how much each one influences the robot's actions.

  • New diffusion policy trains one expert per sensor and blends them via a learned router, avoiding feature concatenation issues. Paper
  • On RLBench, it hits 66% average success versus 56% for concatenation, using just 0.3% more parameters.
  • On real UR5e tasks, RGB+Tactile concatenation collapses to 5% on occluded marker picking; policy consensus reaches 65%.
  • Perturbation analysis shows vision dominates during approach, then tactile takes over on contact, learned automatically.
  • Independently trained RGB and tactile policies can be composed with fixed weights, no joint retraining. Code
  • Dataset and pretrained assets released on Hugging Face; paper appears at ICRA 2026.

Robot manipulation policies have a bad habit: when you feed them vision and touch at the same time, vision wins. A new paper from researchers at Illinois, Harvard, MIT, and Columbia argues that the standard trick of concatenating sensor features into one big vector quietly sabotages contact-rich tasks, and proposes a compositional alternative that trains one diffusion expert per modality and blends them at the policy level.

The work, titled Multi-Modal Manipulation via Multi-Modal Policy Consensus, is set to appear at ICRA 2026. The team has released code and a dataset on Hugging Face, along with a project page containing videos of the robot in action.

Why concatenation quietly breaks contact-rich tasks

Consider a robot trained to retrieve an object from an opaque container. For 90% of the trajectory, vision guides the approach and gripper positioning. Once the gripper enters the container, however, cameras go blind while tactile signals become critical for success.

Feature concatenation struggles with this sparsity: the learning algorithm downweights the rarely-active tactile stream as noise, focusing instead on the statistically dominant visual features. Once trained, the monolithic network is also frozen in place. Swap in a new sensor or lose one at runtime, and you retrain from scratch.

Teaser figure showing modality importance shifts and policy-level composition

One diffusion policy per sense, then a router picks sides

Rather than fusing features early, the method factorizes the policy into modality-specific diffusion models. Each modality (RGB, point cloud, tactile, DINO features, etc.) gets its own set of sub-policies that model the action distribution conditioned only on that sensor. In the experiments, the authors use two sub-policies per modality to capture complementary behavioral modes, such as geometry versus fine detail for vision, or contact onset versus sustained force for tactile.