Stanford's LILIE Cuts Image Generation Error by 13.83 Points Using Captions
Stanford's LILIE encoder gives diffusion models a semantic yardstick instead of per-pixel distance, cutting generative Frechet Distance by 14 points on GPIC.
- Stanford introduces LILIE, a lossless language-informed image encoder that provides a semantic loss for text-to-image diffusion training.
- Combining pixel and LILIE losses cuts gFD on GPIC from 66.84 to 53.01 at 1.1B parameters.
- LILIE is a TARFlow normalizing flow transformer trained jointly with a LoRA-tuned VLM on noise-corrupted features.
- Ablation with random-token captions makes generation worse, confirming semantic content drives the gains.
- Gains hold across short, medium, and long prompts at both 700M and 1.1B generator sizes.
- Code, encoders, and models available at suchiragarwal.com/lilie and Hugging Face.
Pixel loss assigns equal weight to equal-sized errors, whether they distort a subject’s face or an empty patch of sky. Stanford researchers propose LILIE, a Language-Informed and Lossless Image Encoder that uses captions to shape the training loss for text-to-image diffusion models.
On a 1.1-billion-parameter JiT model, replacing pixel loss with LILIE reduced generative Fréchet Distance from 66.84 to 55.78. Combining the two losses reached 53.01, a 13.83-point reduction from the baseline. The generator architecture and inference path remained unchanged.
Equal pixels hide unequal errors
Pixel-space diffusion offers a direct alternative to systems built around learned image tokenizers such as variational autoencoders. Removing the tokenizer simplifies the pipeline and avoids its reconstruction bottleneck, but the model still needs a useful measure of prediction error.
Pixel-level L2 loss measures the squared difference between corresponding pixel values. Its uniform weighting ignores how much each region contributes to the image’s meaning. Two predictions can receive similar losses even when one preserves the captioned subject and the other damages it.
LILIE addresses that semantic mismatch by measuring errors in a feature space trained to preserve caption-relevant information. The diffusion model therefore receives larger penalties when its prediction changes features associated with the image description.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
