CMU's HiT Builds Smarter Image Generators With 2.4x Less Training Compute
A pixel-space diffusion transformer from CMU and Rice learns to merge related image patches by itself. It scores 1.48 FID on ImageNet and needs 2.4× less training compute than JiT to reach the same quality.
- CMU and Rice researchers released HiT, a pixel-space diffusion transformer that learns to merge related patches into hierarchical groups.
- HiT-XL/16 reaches 1.48 FID on ImageNet 256×256, compared with 1.86 for the similarly sized JiT-H/16 baseline.
- At matched quality (FID ≈ 2.16), HiT needs 2.4× less training compute and 2.3× fewer updates than JiT.
- Object and part groups, plus word-to-region links, appear without segmentation masks or region annotations.
- The 250M text-to-image HiT-B/16 edges past UniLiP-3B's five-benchmark average with 6× fewer parameters, but trails on GenEval.
- Code is MIT-licensed on GitHub, with checkpoints on Hugging Face.
Many high-performing image generators use two stages. A pretrained autoencoder first compresses each image into a small latent grid, then a diffusion model learns to generate within that space. Pixel-space models generate directly from RGB values, removing the separately trained visual bottleneck while taking on more computation and a harder optimization problem. Researchers from Carnegie Mellon University and Rice University propose a fix called HiT, short for Hierarchical Diffusion Transformer, in the HiT preprint.
HiT targets a common Transformer design choice: retaining one token per image patch throughout the network. The model instead merges related tokens, processes the compressed representation, and restores the original patch layout. Its learned groups often correspond to objects, object parts, and background regions, despite receiving no segmentation labels.
Raw pixels carry three costs
- More computation: A typical latent model reduces each spatial dimension by 8×. A pixel model operates at full image resolution, and self-attention cost grows quadratically with the token count.
- Slower convergence: The model must learn fine texture and global composition together. A latent model delegates much of the low-level detail to its autoencoder.
- No visual prior: Latent models begin in a representation that already captures image structure. Some systems also align their representations with pretrained encoders such as DINOv2. Raw RGB values provide no corresponding signal about which pixels belong together.
HiT addresses the missing visual prior through compression. To preserve useful information while reducing the token count, the model must infer which patches represent related content. The resulting hierarchy also cuts the cost of the large central Transformer stack.
Group, process, restore
HiT uses an hourglass architecture with two compression levels. Outer Transformer blocks process every patch token. Two successive grouping stages then produce a smaller set of group tokens, which pass through the central stack. Ungrouping stages restore features to their original image positions, while skip connections recover higher-resolution information that could otherwise disappear through the bottleneck.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.