ByteDance's DreamStereo Converts Flat Video to VR Stereo at 25fps
ByteDance's DreamStereo turns monocular clips into stereo 3D at 25 FPS on HD video by pruning 70% of tokens and warping smarter.
Takeaways
- ByteDance's DreamStereo converts mono video to stereo at 25 FPS on HD, single A100.
- Gradient-Aware Parallax Warping uses backward warping plus Jacobian gradients for clean occlusion masks.
- Parallax-Based Dual Projection fabricates stereo training pairs from monocular video, no stereo cameras needed.
- Sparsity-Aware Stereo Inpainting prunes 70%+ of tokens, giving a 10.7x DiT speedup.
- Achieves 30.48 PSNR on HD-100 at single-step inference, beating ProPainter, StereoCrafter, ZeroStereo.
- Built on Wan2.1-1.3B with LoRA plus a distilled 3D VAE, project page live.
Converting a regular 2D video into a stereo pair that looks convincing on a Vision Pro or Quest headset has been stuck in slow, artifact-prone territory. A new system from ByteDance called DreamStereo pushes stereo video inpainting into real time, generating a coherent right-eye view for HD footage at 25 frames per second on a single A100 with a single denoising step, while beating prior diffusion baselines on quality.
The awkward middle of mono-to-stereo
When you re-project a monocular video into a second camera view, most of the frame stays the same. Only thin slivers along object boundaries become newly visible, where the original camera never saw anything. Those slivers, the disocclusions, are what needs to be filled in.
That geometry creates two nasty problems. Existing approaches struggle because high-quality stereo inpainting datasets are scarce, which limits their ability to learn effective priors. They also apply equal processing to every region of the frame even though most pixels require no modification, wasting compute. Running full attention through a diffusion transformer over an entire HD frame just to paint a few edge pixels is enormously wasteful, since attention scales as O(N²) with the number of visual tokens.
Where the field was stuck
Prior work took two main routes, both leaky. Training-free methods like StereoDiffusion and ZeroStereo manipulate latents during denoising but suffer domain gaps and, in video, temporal flicker. Training-based systems like StereoCrafter, ImmersePro and StereoConversion learn the inpainting task properly, but acquiring stereo videos is expensive, and different stereo sources adopt varying binocular projection rules, or even simple pixel translation, which degrades data quality.
TrajectoryCrafter proposed a clever workaround: synthesize training pairs from monocular video by re-projecting it twice. It relied on forward warping, which scatters pixels at object edges, compromising dataset quality and degrading inpainting at inference through incorrect foreground pixels.