Skip to main content
Back

StructDiff Trains on a Single Photo and Beats Every Rival Generator

A single-scale diffusion model uses adaptive receptive fields and 3D positional encoding to preserve structure and enable spatial control from one training image.

Pro

  • StructDiff is a single-image diffusion model accepted to IEEE Transactions on Multimedia.
  • An Adaptive Receptive Field block with 5/7/9/11 kernels replaces multi-scale training and prevents structural drift.
  • 3D positional encoding (x, y, foreground mask) with Fourier embedding enables translation, scaling, and local mask edits.
  • Beats SinGAN/SinDDM/SinFusion/SinDiffusion on SIFID, MUSIQ, DB-CNN, and a new GPT-4o judge.
  • Supports text-guided generation, editing, outpainting, harmonization, and paint-to-image without retraining.
  • Code, dataset, and project page are public: GitHub, HuggingFace.

Training a diffusion model on exactly one image sounds counterintuitive, but that is the whole point of a subfield called single-image generation. You feed the model a single natural photo, it learns the internal patch statistics of that image, and then it can sample diverse variations that look like they came from the same source. The tricky part has always been keeping large rigid structures intact while giving the user any real control over layout. A new framework called StructDiff, accepted to IEEE Transactions on Multimedia, attacks both problems at once.

Why single-image generators keep breaking structure

SinGAN pioneered this area with a pyramid of Patch-GANs that learn hierarchical structure and texture distributions, and later work swapped in diffusion models for stability. Two architectural families have dominated since, and both have a fatal flaw when the source image contains a big rigid object like a face, a car, or an animal.

  • Multi-scale methods (SinGAN, SinDDM) train separate generators at different resolutions. The hierarchical design enables multi-scale perception but introduces error accumulation during coarse-to-fine generation, so shapes drift as you go up the pyramid.
  • Single-scale methods (SinDiffusion, SinFusion) use one generator at fixed resolution. They avoid error accumulation, but the fixed receptive field cannot simultaneously capture fine textures and global structures. Small receptive fields excel at local details and miss large-scale patterns; large ones capture global context but blur fine details.