NVIDIA's Robota-Embed Lets Robots Search Past Experience 2x Better Than Gemini
NVIDIA's Robota-Embed is a vision-language-action retrieval model that pulls useful past experience into robot policies, lifting task success and cutting token costs.
- NVIDIA's Robota-Embed jointly encodes vision, language, and native robot actions in one retrieval space.
- Clip-level R@1 of 47.33% beats Gemini Embedding 2 by 22.09 points across four domains.
- Dropping action from the query hurts retrieval more than dropping vision (20.08% vs 23.00%).
- Embed2ICL lifts LIBERO difficult-subset success from 86.4% to 92.8% via retrieved context.
- Embed2Evolve raises frozen GPT-6 Astra from 24% to 68% success on five ManiSkill tasks.
- Code, weights, and the benchmark are listed as coming soon.
Robota-Embed brings robot actions into multimodal retrieval
Robot fleets collect extensive interaction logs, yet policies have lacked a reliable way to retrieve useful experience during a task. NVIDIA’s new Robota-Embed model encodes images, language, and native robot actions in one embedding space, allowing a policy to search past interactions and use the result as runtime context.
Robota-Embed defines a relevant match as an interaction whose recorded continuation fits the robot’s current state. The paper trains the model to connect a present observation and action with the next four seconds of video and control data.
Actions change the match
Existing robotics retrievers often inherit vision-language encoders such as CLIP or Gemini Embedding. Those models can align images with instructions, but they do not directly represent proprioceptive or control signals that describe the robot’s movement.
Action data contributes more than vision in the paper’s inference-time ablation. Removing actions reduces average clip-level recall at rank one, or R@1, from 47.33% to 20.08%. Removing vision produces 23.00%. R@1 measures how often the correct item appears as the retrieval system’s first result.
| Query modalities | Clip-level R@1 |
|---|---|
| Vision, language, and action | 47.33% |
| Vision and language | 20.08% |
| Language and action | 23.00% |
| Vision and action | 7.10% |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
