---
title: "NVIDIA's PixelUMM Merges Image and Video AI Into One 8B Model"
subtitle: "Researchers from NVIDIA and the University of Waterloo propose a unified multimodal model that reads and writes raw pixels, skipping vision encoders and VAEs entirely."
url: "https://alphasignal.ai/news/nvidia-s-pixelumm-merges-image-and-video-ai-into-one-8b-model"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-09-29T21:59:07.000Z"
updated: "2026-10-05T18:07:47.828Z"
type: "Paper"
topics:
  - "Video"
  - "Image"
subtopics:
  - "Video Generation"
  - "Video Understanding"
  - "Image Generation"
upvotes: 276
source_url: "https://arxiv.org/abs/2609.38597"
comments: 0
truncated: true
---

# NVIDIA's PixelUMM Merges Image and Video AI Into One 8B Model

> Researchers from NVIDIA and the University of Waterloo propose a unified multimodal model that reads and writes raw pixels, skipping vision encoders and VAEs entirely.

Researchers from NVIDIA’s Toronto AI Lab and the University of Waterloo have published PixelUMM, an 8-billion-parameter model for image and video understanding and generation. A single decoder-only Transformer maps pixel patches directly into and out of the model, removing the separate vision encoder used for understanding and the variational autoencoder commonly used for generation.

The research release includes a [project page](https://nv-tlabs.github.io/PixelUMM/), [source code](https://github.com/nv-tlabs/PixelUMM), and an [8B checkpoint](https://huggingface.co/nvidia/PixelUMM). Developers can inspect the architecture, reproduce its experiments, and test the trade-offs documented in the paper.

## The cost of stitched-together vision

Unified multimodal models commonly combine several visual pipelines. A CLIP- or SigLIP-style encoder converts images into semantic features for understanding, while a VAE compresses pixels into a latent space for diffusion-based generation. Some systems add another tokenizer for autoregressive image synthesis. Each component brings its own parameters, interfaces, training behavior, and visual token format.

Earlier pixel-space models showed that Transformers can process images without a dedicated vision encoder. Video adds a harder representation problem: understanding benefits from sparsely sampled frames, while generation needs dense spatiotemporal regions for flow matching or diffusion. PixelUMM reconciles those formats inside one backbone.

## Inside the shared backbone

PixelUMM divides images into 16-by-16-pixel patches. Its video-generation representation uses spatial patches spanning four frames, described as p16/t4\. A Qwen3-8B backbone processes text, clean visual inputs, and noisy generation targets through shared self-attention layers.

The model uses a Mixture-of-Transformers design with separate feed-forward experts for understanding and generation. Shared attention allows every modality to exchange information, while the task-specific experts specialize the intermediate computation.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation](https://arxiv.org/abs/2609.38597)
- [PixelUMM Project Page](https://nv-tlabs.github.io/PixelUMM/)
- [PixelUMM Code Repository](https://github.com/nv-tlabs/PixelUMM)
- [PixelUMM Model Weights](https://huggingface.co/nvidia/PixelUMM)
