Reka's WMGym Judges World Models on Decision-Making, Not Pretty Pixels
Reka's WorldModelGym shifts world model evaluation from visual realism to decision-making accuracy, launching an open leaderboard with a simple API submission protocol
- New leaderboard: Reka launched WorldModelGym (WMGym), evaluating world models on decision fidelity rather than visual realism.
- Core metric: Decision fidelity measures how often a world model correctly ranks which action leads to the best outcome.
- Four benchmarks: Classical Control, DeepMind Control, Meta-World, and Atari form the four evaluation families.
- Baseline results: Reka's Dreamer-v3 leads three of four families, hitting 84% decision fidelity on Classical Control.
- Open submission: Anyone can enter by hosting a
/scoreendpoint; Reka calls it and computes the score — no weight uploads needed. - Free to use: The leaderboard is open and free; the benchmark targets robotics, game AI, and simulation-based planning use cases.
Most world model benchmarks ask one question: does the generated video look real? WorldModelGym (WMGym) from Reka asks something different: given a set of possible actions, can the model correctly predict which one leads to the best outcome? That shift targets the gap between a model that generates convincing pixels and one that actually understands how the world works.
The leaderboard is now open. Reka seeded it with their own Dreamer-v3 implementation as the first entry and is inviting the community to submit models via a lightweight API protocol.
The problem with looking good
The world model field has been dominated by video generation quality metrics: does the output look photorealistic, does motion flow naturally, does physics seem plausible to a human eye? Existing benchmarks treat world models as video generators and ignore their functional role in decision-making. The result is a leaderboard ecosystem that rewards perceptual polish but says nothing about whether a model can serve as a simulator for an RL agent or a planning system.
WMGym addresses that gap directly. Its core metric is decision fidelity: how often does the world model rank actions in the same order as the ground-truth environment would? If a model predicts that pushing left leads to a higher reward than pushing right, and the real environment agrees, that counts as a fidelity hit.