Skip to main content
Back

UT Dallas's iTeach Lets Robots Fix Their Own Perception in 25 Minutes

A framework lets robots learn from their own mistakes during deployment, using human gaze, voice, and short object interactions to fix perception failures on the fly.

  • iTeach adapts robot perception during deployment using a co-located human, HoloLens 2, and short object interactions.
  • A Few-Shot Semi-Supervised labeling strategy annotates only the final frame via gaze and voice, then propagates with SAM2.
  • Iterative fine-tuning improved MSMFormer's combined UOIS score from 26.1 to 80.7 on real-world scenes.
  • Performance saturates around 20 collected scenes, showing data diversity matters more than volume.
  • Swapping in iTeach-UOIS on SceneReplica lifted pick-and-place success by 7 points over MSMFormer.
  • Full failure-to-redeployment loop takes 15-25 minutes; code and data are public.

Robotic perception models tend to look great in the lab and stumble the moment they encounter a real living room. Clutter, weird lighting, and objects the model has never seen before break segmentation and detection in ways that ripple through the entire manipulation stack. A team at UT Dallas has proposed a workflow called iTeach that skips the usual retrain-and-redeploy cycle and instead lets a human standing next to the robot fix perception failures within minutes.

The system pairs a Fetch mobile manipulator, a Microsoft HoloLens 2 headset that overlays perception outputs in the user's field of view and captures hands-free gaze and voice annotations, and a compute node that handles inference, dataset aggregation, and fine-tuning. Together they form a closed loop where failures are spotted during deployment, targeted supervision is generated through short human-object interaction, and the model is refined and redeployed within a single session.

Why perception keeps breaking in the wild

Large pre-trained vision models show strong zero-shot generalization on benchmarks, but in real environments they fail under out-of-distribution conditions like clutter, novel object instances, lighting variation, and occlusions. Those failures are the primary bottleneck for real-world autonomy because perceptual errors propagate straight into downstream decisions and task execution.

The standard fix is to collect more data, annotate it offline, retrain, and redeploy. That pipeline can take hours or days per iteration, and static dataset expansion cannot cover the highly context-specific long-tail failures that actually show up in deployment. Prior human-in-the-loop work like DAgger, HG-DAgger, and HIL-SERL mostly operates at the action level, correcting policies rather than the perception module that caused the mistake in the first place.

The HumanPlay trick

The core idea is disarmingly simple. When the robot's model produces a bad segmentation, the human physically rearranges the scene to make it easier to label, while the system records the whole RGB-D sequence.