Ai2's MolmoMotion Predicts 3D Object Motion Before It Happens, Beating Video AI

AI models have gotten remarkably good at tracking how things move in video. But tracking is retrospective , it tells you where something went, not where it's going. MolmoMotion, a new open model from the Allen Institute for AI (Ai2), flips that around: given a video frame, a set of 3D points on an object, and a plain-language instruction like "Put the white bowl on the table," it predicts where those points will travel over the next few seconds in real-world 3D space.

The release is a full stack: the model weights, a 1.16-million-video training dataset called MolmoMotion-1M, and a new evaluation benchmark called PointMotionBench. Everything is openly available on Hugging Face.

The problem nobody had solved cleanly

Motion forecasting , anticipating how objects will move before they move , is a surprisingly unsolved problem. Prior approaches fell into one of three buckets, each with a fatal flaw:

• Pixel-space video generators (like Wan2.2 or Cosmos Predict) generate plausible-looking future frames, but visually plausible video doesn't mean the predicted motion is metrically accurate. They spend enormous compute rendering appearance when you only need geometry.
• Parametric 3D models (pose estimators for human bodies, hands, or rigid objects) are accurate but only work for specific object categories they were designed for.
• 2D point trajectory methods are category-agnostic, but 2D image-plane coordinates mix object motion with camera movement, making them hard to use downstream.

MolmoMotion's answer is to represent motion as object-attached 3D points in a shared world coordinate frame. This representation is class-agnostic (works on any object), view-stable (the same motion looks the same regardless of camera angle), and compact enough to pass directly into downstream systems like robot planners or video generators.

Two models in one

MolmoMotion uses Molmo 2 as its backbone, allowing it to connect language instructions to objects and points in an image. The team trained two complementary variants:

• MolmoMotion-AR (autoregressive): Predicts future coordinates step by step, representing 3D coordinates as structured text and writing out the future trajectory in temporal order. Because each new coordinate is conditioned on the trajectory already generated, this encourages smooth rollouts and gives the strongest accuracy when the future path is well-defined.
• MolmoMotion-FM (flow-matching): Predicts trajectories in continuous 3D space by transforming noise into motion, which makes it better suited for representing uncertainty when an instruction admits multiple plausible futures. It's also roughly 150x faster at inference , 1.1 seconds per clip vs. 148 seconds for AR at T=32 on a single A100.

The autoregressive variant wins on accuracy benchmarks; the flow-matching variant is what you'd reach for in real-time robotics or video generation pipelines.

Training data that didn't exist

To train MolmoMotion, Ai2 needed data that didn't yet exist: large-scale videos with 3D point trajectories grounded to specific objects and paired with action descriptions. Existing 3D-track datasets were small and domain-limited, and while internet videos have all the scale and diversity needed, they didn't include 3D annotations.

So they built an automated pipeline to extract this data from ordinary video. The pipeline runs in five stages:

1. Object grounding: An LLM extracts the moving object from the action description; MolmoPoint localizes it as a 2D point; SAM 3 segments it.
2. Query point sampling: K-means cluster centers spread 100 points across the object surface.
3. 2D tracking: AllTracker propagates those points through the video with persistent 2D tracks.
4. 3D lifting: ViPE estimates per-frame metric depth and camera geometry, back-projecting 2D tracks into a shared world frame in physical units (meters).
5. Filtering and clipping: Noisy trajectories are smoothed; clips are trimmed to windows where the object actually moves.

Running this pipeline at scale yielded MolmoMotion-1M , to Ai2's knowledge the largest corpus of action-described, object-grounded 3D point trajectories assembled to date, spanning 736 motion types and 5.6K distinct objects. Source videos come from EgoDex, HD-EPIC, Xperience-10M, MolmoSpaces, DROID, YT-VIS, and Stereo4D , covering egocentric human manipulation, robot manipulation, and outdoor in-the-wild scenes.

Beating every baseline on a new benchmark

To evaluate MolmoMotion's forecasting performance, Ai2 also built PointMotionBench, a human-validated benchmark of held-out 3D trajectories covering 2.7K clips spanning 111 object categories and 61 motion types, including indoor manipulation, egocentric hand-object interaction, and outdoor dynamic scenes.

On PointMotionBench, MolmoMotion outperforms all existing 3D motion forecasting methods tested , including pixel-space video generators, parametric 3D methods, and a simple constant-velocity baseline , across a range of objects, scenes, and actions. One particularly telling result: simple non-parametric baselines (just holding an object still, or extrapolating its current velocity) beat several learned methods including Wan2.2-5B and Cosmos Predict, suggesting that generating visually plausible video doesn't actually recover accurate metric motion.

What it actually unlocks downstream

The real bet here is that a good 3D motion prior is transferable. A human hand and a robot gripper move very differently, but the cup they're both picking up follows roughly the same arc through 3D space. That insight drives two concrete downstream applications.

Robotics planning: After fine-tuning on DROID, a control policy built on MolmoMotion succeeds on 76.3% of pick-and-place tasks versus 56.0% for the same policy built on Molmo 2 , and it learns faster, reaching 51% after 10K training steps where the Molmo 2 version tops out at 19%. The gains hold even for unseen objects in unseen scenes (74.2% vs. 48.7%), which is the hard generalization case that matters for real deployment.

Video generation: MolmoMotion's predicted paths can steer video generation. Instead of letting an image-to-video model guess motion from a text instruction alone, feeding in MolmoMotion's predictions results in generated video that follows requested actions more closely, especially for small and precise movements a prompt can only describe vaguely. When used to guide a video generator, MolmoMotion improves motion quality over the base model on all five motion-related metrics measured, and beats a much larger image-to-video model on four of the five.

What it can't do yet

MolmoMotion uses eight query points per object during training , enough to forecast a useful trajectory but not enough to densely represent surface geometry. This limits the model's handling of complex deformable motion. The autoregressive variant is also slow: at 148 seconds per clip for T=32 on an A100, it's not suitable for real-time use without switching to the flow-matching variant. And while real-robot trajectory prediction looks promising after fine-tuning on DROID, closed-loop real-robot evaluation is left to future work.

The bigger picture

The field has implicitly assumed that perception , understanding what's already there , is the foundation of visual intelligence. MolmoMotion is a direct challenge to that assumption. Ai2 argues that forecasting , anticipating how objects in the world will move before they move , is as fundamental to machine intelligence as perceiving what's already there.

The practical implication is significant: if a motion prior learned from ordinary human video can transfer to robot manipulation with a 20-percentage-point improvement in success rate, then the vast corpus of internet video becomes a training resource for robotics , not just for perception, but for physical reasoning about what happens next.

All weights, training data, and the PointMotionBench benchmark are available now. The model collection and MolmoMotion-1M dataset are on Hugging Face, with the full technical paper also publicly available.