Kyutai's MIRA Runs Playable Rocket League Inside a 5B Neural Network

A joint team from General Intuition, Kyutai, and Epic Games has released MIRA, a playable neural network that simulates four-player Rocket League matches purely from pixels and controller inputs. MIRA is a real-time world model of Rocket League: a 5B parameters latent diffusion model generates the video frame by frame from all four players' actions, so a full 2v2 match can be played inside the model at 20 FPS on a single GPU. There is no physics engine, no renderer, and no 3D scene under the hood at inference time. The model is the game.

You can try the browser demo now at mira-wm.com, and the training and inference code sit on GitHub under Apache 2.0. The 1,000 hour Rocket Science dataset is on the Hugging Face Hub under CC BY-NC-SA 4.0.

A neural game engine that answers to your keyboard

MIRA is a 5-billion-parameter latent diffusion world model designed for highly dynamic, multi-agent environments. Using the competitive video game Rocket League as its testbed, it simulates four-player matches by conditioning on concurrent, multi-agent action streams, allowing it to predict future game states in real time (20 frames per second on a single Nvidia B200 GPU). The system takes a stream of key presses from up to four players and generates the next video frames on the fly, so latency stays low enough to actually play.

To be clear on scope: the model runs on a single (beefy - B200) GPU, producing 576p video at 20fps that's controllable in real time. The demo streams the result to your browser, and Epic Games explicitly gave permission for the gameplay recordings but did not use MIRA to develop the actual game.

Why pretrained vision features change the math

The core technical bet is that you should not learn a video codec from scratch when a strong vision backbone already exists. Most video world models pass frames through a blockwise causal VAE trained for the task. MIRA takes a different route by extending the Representation Autoencoder (RAE) framework from images to video.

The codec itself is a video autoencoder built on a frozen DINOv3-L feature extractor. The key finding: if you build the latent space on top of a pretrained feature extractor rather than learning it from scratch, then predicting in that space becomes noticeably easier and more stable over long horizons. The feature extractor stays frozen; only a linear bottleneck and the decoder are trained. A learned linear bottleneck downsamples these features (a 2x2 spatial and 2x temporal reduction), while a space-time causal Vision Transformer (ViT) decoder reconstructs the video frames.

The intuition, spelled out in the technical report discussion, is that an encoder trained from scratch might achieve sharper single-frame reconstructions, but its latent space is 'fragile.' DINOv3 features live in a semantically structured space, so when the diffusion transformer predicts a slightly wrong latent, decoded frames still look like Rocket League instead of drifting into noise.

Fitting four players into one prediction

Multiplayer is the harder problem. Each player sees the arena from their own camera, but the ball, cars, and score have to stay consistent across all four views. MIRA solves this with a spatial stitching trick and a training strategy borrowed from dropout.

• Its multi-perspective scheme stitches together the views of four players into a unified grid to enhance spatial consistency across perspectives.
• The action dropout mechanism introduced during training completes the behavior of vehicles not under command when part of the action stream is missing.
• This action dropout strategy acts as a strong auto-pilot mode, allowing MIRA to handle anywhere from 1 to 4 players.

In practice this means you can drop into a match alone and the model will hallucinate plausible teammates and opponents, or you can join with three friends and each player only controls their own car.

Ten thousand hours of bot-versus-bot

MIRA is trained on 10k hours of synthetic Rocket League data collected by General Intuition using four instances of the publicly available Nexto bot playing against each other, while recording their actions. Because the data comes from bots hitting keys, every frame has clean action labels aligned to it, which is exactly what a controllable world model needs.

The publicly released subset gives outside researchers a real handle on the problem. The team has open-sourced both the training and inference code, and released the Rocket Science dataset, which includes 1,000 hours of matches and approximately 4,000 hours of video, motion flow, and physics state data across four perspectives. Each sample bundles synchronized 4-second windows from all four players with per-frame keyboard state, physics, and game events.

Loading a clip looks like this:

from mira.data import RocketScienceDataset
ds = RocketScienceDataset.from_hub("kyutai/rocket-science", split="test", shards=1)
clips = ds.load_match(ds.match_ids()[0], clip_len=16, target_fps=20)
clip = clips[0]
clip.frames    # (P=4, T=16, C, H, W) uint8 video
clip.actions   # (P, T, 9) int32 multi-hot keyboard state
clip.physics   # ball, cars, score per player per frame

If you want to retrain the codec yourself, note that codec training uses a frozen DINOv3-L/16 encoder whose weights are gated by Meta. Download dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth from the DINOv3 page and set RS_DINO_WEIGHTS_DIR=/path/to/weights. World-model training and inference don't need it.

What it is actually good and bad at

The demo video looks strikingly close to real Rocket League on quick glance, but MIRA has clear limits. It runs at 576p and 20 FPS, well below the 60+ FPS competitive players expect. The horizon is short, and long term drift is exactly the failure mode the DINOv3 anchoring is designed to fight, so extended rallies still stress the model. It also only knows what Nexto bots do, so exotic human strategies are outside its training distribution.

Where it shines is as a controllable, differentiable stand-in for a full game loop. Every frame is a function of actions you can backpropagate through in principle, which opens the door to training agents entirely inside the simulator.

Why simulate a game you can already play

The honest answer from one of the authors: the project is not useful in and of itself. It's research on a more controlled environment that should help with doing world models for physical AI later - think self-driving and robots. Rocket League is a stress test. It has fast physics, four agents interacting, and easily collectible ground truth from bots. If you can build a neural simulator that stays coherent here at 20 FPS, the same recipe should transfer to driving scenes and manipulation.

In the broader landscape, MIRA sits alongside work like Wayve's GAIA-2 for driving and Microsoft's MineWorld for Minecraft. What sets it apart is the combination of multiple concurrent agents, real time interactivity, and the RAE-on-video architectural choice. For anyone building world models, the practical takeaway is that swapping a bespoke VAE for a frozen DINOv3 encoder plus a light bottleneck is worth trying before scaling parameters. The code, weights recipe, and dataset are all now on the table to test that claim.