Physical Intelligence and NVIDIA's SC3-Eval Replaces Real Robot Testing With AI

Testing a robot policy in the real world is slow, expensive, and brutally hard to scale. A human has to set up the scene, watch the robot attempt the task, reset everything, and repeat -- dozens of times per policy checkpoint. While recent years have seen substantial progress in developing more capable and general robot policies, their evaluation remains a persistent challenge, and the problem becomes especially acute as policies grow more generalist, requiring broader and more diverse evaluation scenarios. Now, SC3-Eval, a joint project from Physical Intelligence and NVIDIA, proposes a different answer: skip the real world entirely and let a video world model do the judging.

The Evaluation Bottleneck Nobody Talks About

Evaluating generalist robot manipulation policies in the real world is expensive, slow, and difficult to scale. As robot foundation models -- the VLAs (vision-language-action models) that take camera feeds and language instructions and output motor commands -- become more capable, the number of tasks they need to be tested on grows with them. Physical Intelligence noted that evaluating all the different things their models can do takes longer and longer, and a world model eval that needs only a couple hours of GPU time is a compelling alternative.

Real-world evaluation is inherently unscalable: it is limited by logistics, safety concerns, and reproducibility issues, and requires significant human involvement for setup, execution, and scoring. Human operators must supervise trials and manually reset scenes, which restricts the scale and frequency of evaluations. The idea of using a learned world model as a proxy evaluator has been explored before, but prior work struggled with a core technical problem: errors compound.

When a video world model generates the next frame from the current frame and the robot's action, small mistakes accumulate. By step 50 of a manipulation task, the imagined scene can look nothing like reality. This is the autoregressive drift problem -- each generated frame is fed back as input for the next prediction, so any hallucination snowballs.

Three Consistency Tricks That Make It Work

SC3-Eval is a self-consistent video generation recipe that adapts a pre-trained video foundation model into an accurate policy evaluator by enforcing three complementary forms of consistency. Each one attacks a different failure mode.

1. Forward-inverse dynamics consistency. Forward-inverse dynamics consistency jointly trains the model to predict frames from actions and to recover actions from frames, anchoring generated rollouts to a physically plausible action manifold and counteracting the drift a forward-only model cannot penalize. Think of it as a two-way check: the model must be able to both predict what happens given an action, and infer what action must have caused a given transition. If these two directions disagree, the training signal corrects the model.
2. Cross-view consistency. Real robots use multiple cameras -- an external view and a wrist-mounted camera. Cross-view consistency trains the model to inpaint each camera view from the other, keeping the multi-camera observation coherent over long rollouts without any explicit memory mechanism. Without this, the two camera feeds can drift into contradictory states.
3. Test-time consistency. Test-time consistency reuses the inverse dynamics mode at inference as a per-action-chunk uncertainty signal that terminates rollouts whose generated frames drift away from the requested actions. In other words, the model knows when to quit: if the imagined frames stop making physical sense relative to the actions being taken, the rollout is halted rather than continuing to hallucinate.

The world model used is built on NVIDIA's Cosmos video foundation model, fine-tuned on Physical Intelligence's robot data. The VLA policy runs in a closed loop: it sees the world model's generated frames, outputs actions, and the world model predicts what happens next -- no real robot required.

The Numbers Are Hard to Argue With

Across seven real-world vision-language-action policies, SC3-Eval attains a closed-loop Pearson correlation of 0.929 and MMRV of 0.119, outperforming three strong prior video-model-based baselines, and generalizes to new tasks. A Pearson correlation of 0.929 means the world model's predicted success rate is nearly a perfect linear predictor of the actual success rate you'd measure by running the robot. MMRV (mean metric rank violation) measures how often the model gets the relative ranking of two policies wrong -- 0.119 is a low error rate.

The benchmark used is a table bussing task -- a robot arm clearing dishes from a table into bins, evaluated across seven VLA checkpoints of varying quality. SC3-Eval tracks real-world policy success rates across in-distribution and out-of-distribution table bussing tasks, including fine-grained success criteria. Success is measured along three axes: language following (did the robot pick the right object?), object lifting, and object placing. The model was also tested on a reversed version of the task -- moving objects back -- which it had never seen during training, and still generalized.

One of the most striking findings: physical failures transfer. A policy that fumbles a grasp in the real world fumbles it in the world model too. SC3-Eval rollouts reproduce the failure modes that policies exhibit in real-world rollouts, supporting fine-grained diagnostic comparison rather than aggregate ranking alone. This means you can use the world model not just to rank policies, but to debug them.

What This Unlocks in Practice

The practical implications are significant for anyone training robot foundation models at scale. Instead of running hundreds of physical trials every time a checkpoint is saved, teams can:

• Run world model evaluations in a couple of hours of GPU time instead of days of robot time
• Evaluate many more checkpoints during training, enabling tighter feedback loops
• Test on tasks that would be dangerous or impractical to run physically at high frequency
• Get qualitative failure analysis, not just a success rate number
• Generalize to new tasks without collecting new real-world evaluation data

The uncertainty estimation is particularly useful. The inverse dynamics model doubles as a confidence meter at inference time -- when the generated frames start diverging from what the requested actions should produce, the system flags the rollout as unreliable and stops early rather than producing a misleading score.

Limitations and What Comes Next

The system is not a complete replacement for real-world evaluation. The world model was trained on Physical Intelligence's own robot data, which means it works best for tasks and embodiments similar to what it has seen. Highly novel tasks or new robot morphologies would require retraining or fine-tuning the world model itself. A key challenge is generating accurate policy videos from world models that faithfully reflect the robot actions, and this remains an active research problem across the field.

No code or model weights have been released alongside the paper, but the project page includes video comparisons of real vs. imagined rollouts, ablations showing the contribution of each consistency component, and uncertainty visualizations. The work sits at the intersection of two fast-moving trends: the scaling of generalist robot policies and the maturation of video world models as physical simulators -- a combination that is quickly making the "just test it in the real world" approach feel like the slow path.