Skip to main content
Back

Meituan's LARY Benchmark Shows Generic Vision Models Beat Robot-Trained AI

A new benchmark shows general vision models like DINOv3 and V-JEPA 2 beat purpose-built robot models at extracting actions from video.

Pro

  • Researchers introduced LARY, a benchmark evaluating latent action representations for Vision-Language-Action models.
  • Dataset spans 1.2M videos, 620K image pairs, 595K trajectories across 151 actions and 11 robot embodiments.
  • General vision encoders like DINOv3 and V-JEPA 2 outperform specialized embodied latent action models.
  • Semantic latent spaces align better to physical action than pixel-based VAEs like FLUX.2 or Wan2.2.
  • Attention analysis shows self-supervised models focus on interaction points; embodied LAMs produce diffuse blobs.
  • Full dataset and code available on Hugging Face and GitHub.

A team of researchers has released LARY, a benchmark that asks a pointed question about the current wave of Vision-Language-Action (VLA) models for robotics: are the specialized latent action encoders being built for embodied AI actually better than off-the-shelf vision models? After evaluating 11 systems on more than a million videos, the answer is uncomfortable for the field. The paper, from Dujun Nie and collaborators, shows that generic visual foundation models trained with no action data at all consistently beat models purpose-built for robot control.

The data island problem

VLA models want to map pixels and language to motor commands, but there is a chronic shortage of labeled robot data. Human action videos on the internet are the obvious substitute, except they arrive without action labels. The paper's proposal is to lean on scalable, unlabeled human video and convert the visual signal into "latent actions": ontology-independent representations that capture motion.

A latent action here means a learned vector describing what changed between two frames, independent of which body did it. Extract that reliably, and you can pretrain a policy on YouTube-scale video and only fine-tune on real robot data. The catch is that nobody had cleanly measured whether these latent representations actually preserve enough physical information to drive a robot arm.

What LARY actually measures

LARY probes representations along two axes that map to a plain-English split: high-level semantic action (what to do) and low-level physical dynamics (how to do it). The first is a classification task: can a probe read the latent and say "pouring" vs "stacking"? The second is a regression task: can it predict the actual end-effector trajectory in continuous space?