Skip to main content
Back

NVIDIA's PixelDiT Beats FLUX at 1024px Without a VAE Bottleneck

NVIDIA's PixelDiT ditches the VAE entirely, generating images directly in pixel space and hitting state-of-the-art quality with a novel dual-level transformer architecture

  • CVPR 2026 Oral & Best Paper Finalist: NVIDIA's PixelDiT generates images directly in pixel space, eliminating the VAE bottleneck entirely.
  • State-of-the-art pixel-space FID: Achieves 1.61 FID on ImageNet 256×256 with 797M parameters and only 311 GFLOPs — far more efficient than prior pixel models.
  • Dual-level architecture: A patch-level DiT handles global semantics; a pixel-level DiT with Pixel Token Compaction and Pixel-wise AdaLN handles texture details efficiently.
  • Scales to 1024×1024 text-to-image: Scores 0.74 GenEval and 83.5 DPG-bench, matching DALL-E 3 and beating SDXL with a 1.3B model at 0.33 samples/sec on one A100.
  • Superior image editing: No VAE means no reconstruction artifacts — fine text and textures survive edits that corrupt FLUX and SD3 outputs.
  • Code and models released: Available on GitHub and Hugging Face for non-commercial research use.

Every major image generation model you've used , Stable Diffusion, FLUX, DALL-E 3 , shares a hidden bottleneck: before any diffusion happens, a separate neural network called a VAE (Variational Autoencoder) compresses the image into a compact "latent" representation. It's a practical shortcut that makes training feasible, but it comes with a cost. That compression is lossy, and the errors it introduces ripple through the entire generation pipeline. NVIDIA Research's PixelDiT asks a simple but radical question: what if you just skipped the VAE entirely?

The VAE Problem Nobody Talks About

Latent Diffusion Models (LDMs) have dominated image generation for years. The standard recipe is: train a VAE to compress images into a smaller latent space, then train a diffusion model to operate in that latent space. It's efficient, but it creates two structural problems that are easy to overlook.

  • Lossy reconstruction: Latent-space modeling has been the standard for Diffusion Transformers (DiTs). However, it relies on a two-stage pipeline where the pretrained autoencoder introduces lossy reconstruction, leading to error accumulation while hindering joint optimization.
  • Objective mismatch: The VAE is trained to reconstruct images well, not to make the diffusion model's job easier. These are different goals, and the gap between them introduces distribution shift that the diffusion model has to compensate for.
  • Fine detail destruction: High-frequency content like text in a scene, fine textures, and sharp edges are exactly what VAE compression tends to smear. Even a perfect diffusion model can't recover what the VAE already destroyed.

The practical consequence shows up most painfully in image editing. If you use a flow-based editing method like FlowEdit on FLUX or Stable Diffusion 3 and ask it to change a bicycle to a motorcycle in a scene with text on a wall, the VAE will have already garbled that wall text before the diffusion model even sees it. The edit is correct, but the background is corrupted.

Going Back to Pixels , But Smarter

PixelDiT is a single-stage, end-to-end model that eliminates the need for the autoencoder and learns the diffusion process directly in the pixel space. Pixel-space diffusion isn't a new idea , ADM (the model behind early DALL-E research) operated in pixels , but it was abandoned because the computational cost is brutal. Attention over raw pixels scales quadratically with image resolution, making megapixel-scale training essentially impossible with naive approaches.

The key insight in PixelDiT is that the problem isn't pixel space itself , it's the lack of an efficient architecture for it. The team's analysis reveals that effective pixel-level token modeling is essential to the success of pixel diffusion. Prior pixel-space models either used aggressive patchification (fast but loses detail) or near-pixel granularity (faithful but computationally catastrophic). PixelDiT introduces a third path.

The Dual-Level Architecture

PixelDiT adopts a fully transformer-based architecture shaped by a dual-level design: a patch-level DiT that captures global semantics and a pixel-level DiT that refines texture details, enabling efficient training of a pixel-space diffusion model while preserving fine details. Think of it as a division of labor: one pathway handles the big picture, the other handles the fine print.

The two pathways work together through two key mechanisms:

  • Pixel-wise AdaLN (Adaptive Layer Normalization): Standard conditioning in transformers broadcasts the same modulation parameters to every token. PixelDiT instead computes independent scale, shift, and gating parameters for each individual pixel, derived from the semantic tokens produced by the patch-level pathway. This lets the global context steer per-pixel updates in a spatially aware way.
  • Pixel Token Compaction: Rather than running full attention over all H×W pixels (which would require 82,247 GFLOPs at 256×256 , completely infeasible), PixelDiT compresses each p×p block of pixel tokens into a single compact token before attention, then expands them back afterward. With p=16, this gives a 256× reduction in attention sequence length. Crucially, unlike a VAE, this compression is temporary and lossless in practice , high-frequency information is preserved through residual connections.

For text-to-image generation, the patch-level pathway is extended with MM-DiT blocks (the same multi-modal design used in Stable Diffusion 3) that fuse text and image tokens. Multi-stage training (512px to 1024px) and inference for text-to-image generation is used to scale up to megapixel resolution. The text encoder is a frozen Gemma-2 model, and the pixel-level pathway receives no text tokens directly , it only sees the semantic context passed down from the patch pathway.

Step-by-Step: How a Forward Pass Works

  1. The raw pixel image is split into non-overlapping p×p patches and projected into patch tokens.
  2. The patch-level DiT runs N transformer blocks with full attention over the (short) patch token sequence, building a rich semantic representation with 2D Rotary Position Embeddings (RoPE).
  3. The semantic tokens from step 2 become the conditioning signal for the pixel pathway.
  4. Each pixel is embedded into a low-dimensional token (hidden size 16, vs. 1152 for the patch pathway).
  5. Pixel-wise AdaLN uses an MLP to expand each semantic token into p² sets of per-pixel modulation parameters.
  6. Pixel Token Compaction squeezes each p×p pixel block into one token, runs global attention, then expands back.
  7. The final pixel tokens are decoded into the predicted velocity (using Rectified Flow training), and the process iterates over 100 denoising steps at inference.

The Numbers

The headline result: PixelDiT achieves 1.61 FID on ImageNet 256 and 1.81 FID on ImageNet 512, surpassing existing pixel generative models. FID (Fréchet Inception Distance) measures how close generated images are to real ones , lower is better. For context, the previous best pixel-space model (JiT-G) scored 1.82 with a 2B parameter model. PixelDiT-XL beats it with 797M parameters and only 311 GFLOPs per forward pass , comparable to latent models like DiT-XL (238 GFLOPs), and a fraction of PixelFlow's 5,818 GFLOPs.

On text-to-image benchmarks: it achieves 0.74 on GenEval and 83.5 on DPG-bench, approaching the best latent diffusion models. At 1024×1024, it outperforms SDXL, Playground v2.5, and Hunyuan-DiT on GenEval, and matches DALL-E 3 on DPG-bench , all with a 1.3B parameter model versus FLUX-dev's 12B. Throughput at 1024px is 0.33 samples/second on a single A100, versus FLUX-dev's 0.04.

ModelSpaceParamsGFLOPsFID (ImageNet 256) ↓
DiT-XLLatent675M2382.27
REPA (SiT-XL)Latent675M2381.42
PixelFlow-XLPixel677M58181.98
JiT-GPixel2B7661.82
PixelDiT-XLPixel797M3111.61

What This Actually Unlocks

To the team's knowledge, this represents a significant breakthrough, as prior pixel-space models have struggled to scale effectively to such resolutions. Getting pixel-space diffusion to work at 1024×1024 has been an open problem. The combination of dual-level architecture, pixel token compaction, and pixel-wise AdaLN is what makes it tractable.

The most immediately practical benefit is for image editing. Because PixelDiT never passes through a VAE, there are no reconstruction artifacts to corrupt the input. Plugging it into FlowEdit (a training-free editing method) preserves fine scene text and textures that FLUX and SD3 both destroy. This is a real advantage for any application where faithfulness to the original image matters: inpainting, local edits, style transfer on detailed images.

Other use-cases where PixelDiT's pixel-native approach shines:

  • Text rendering in images: Latent models notoriously smear text; pixel-space models don't have this bottleneck.
  • High-frequency texture generation: Fabric, skin, fur, and material textures that VAEs tend to smooth over.
  • Faithful reconstruction pipelines: Any workflow where you need to edit and recover an image without introducing compression artifacts.
  • Research into end-to-end generative modeling: A clean baseline for studying generation without the confound of a separately trained encoder.

Limitations and What's Still Hard

Pixel-space diffusion incurs higher computational costs than latent approaches due to the raw data dimensionality, even as PixelDiT narrows this efficiency gap. The model still can't fully match the best latent models on FID , RAE-DiT achieves 1.13 FID versus PixelDiT's 1.61. And the T2I model (1.3B parameters) has a known weakness: the final training outcome may have a slight gap compared to a continuous run without intermediate resumes. The paper also notes that the model sometimes struggles with geometrically complex objects like human hands and intricate architectural scenes, attributed to limited model capacity and training data.

Code, Models, and Access

The full codebase is available on GitHub under NVlabs, covering both class-conditioned (ImageNet) and text-to-image variants. Pretrained weights are released on Hugging Face: nvidia/PixelDiT-ImageNet for the ImageNet models and nvidia/PixelDiT-1300M-1024px for the T2I model. The work and any derivative works may only be used for non-commercial (research or evaluation) purposes under the NSCLv1 License. The paper was accepted as a CVPR 2026 Oral , one of the most selective designations at the conference , and was also named a best paper finalist.

For the generation research community, PixelDiT is a strong signal that the latent diffusion paradigm isn't the only viable path to high-quality image synthesis. The VAE has always been a pragmatic compromise, not a principled choice. With the right architecture, you can do without it , and gain meaningful capabilities in return.