---
title: "U-JiT Beats Latent Video Models by Generating Directly in Pixel Space"
subtitle: "A new UNet-style transformer called U-JiT generates video directly from pixels, beating latent diffusion on Kinetics-600 without needing a pretrained autoencoder."
url: "https://alphasignal.ai/news/u-jit-beats-latent-video-models-by-generating-directly-in-pixel-space"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-11-03T12:11:03.000Z"
updated: "2026-11-05T01:49:10.902Z"
type: "Paper"
topics:
  - "Video"
  - "Benchmarks"
subtopics:
  - "Video Generation"
upvotes: 196
source_url: "https://arxiv.org/abs/2611.04764"
comments: 0
truncated: true
---

# U-JiT Beats Latent Video Models by Generating Directly in Pixel Space

> A new UNet-style transformer called U-JiT generates video directly from pixels, beating latent diffusion on Kinetics-600 without needing a pretrained autoencoder.

Most leading video generators compress clips with a pretrained autoencoder, then train a diffusion or flow-matching model in the resulting latent space. A paper by Théophane Vallaeys, Jakob Verbeek, and Matthieu Cord presents [U-JiT](https://arxiv.org/abs/2611.04764), a model that generates video directly in pixel space and outperforms latent and pixel-space baselines on Kinetics-600.

Video models adopted latent generation because raw clips contain millions of pixel values. Compression can reduce that workload by a factor of 64 or more, making attention and denoising substantially cheaper. The resulting pipeline has two separately optimized components: an autoencoder trained for compression and reconstruction, followed by a generator trained on encoded clips.

That division introduces trade-offs. The autoencoder discards some visual information, and reconstruction quality does not always predict generation quality. Researchers must also choose a compression rate that balances detail, training cost, and the difficulty of modeling the latent representation.

## Why pixels tend to blur

Pixel-space image models such as JiT, short for Just image Transformers, have narrowed the gap with latent image generators. Video presents a larger token problem because every frame adds another spatial grid. Keeping the sequence manageable requires larger spatiotemporal patches, which merge fine texture and motion into coarse tokens.

A fixed-width transformer struggles to recover the high-frequency information lost through that patching process. The paper links this limitation to blurred, oversmoothed frames and a gap between the principal-component spectra of generated and real clips. That spectrum measures how visual variation is distributed across the dataset.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Pixel-space video generation with U-JiT](https://arxiv.org/abs/2611.04764)
- [U-JiT HTML paper](https://arxiv.org/html/2611.04764v1)
- [Author thread on U-JiT](https://twitter.com/webalorn/status/2117999480083562818)
