CPAM Cuts Background Distortion in Half Without Fine-Tuning Stable Diffusion
Researchers from University of Science (VNU-HCM), Monash, and Dayton unveil a training-free image editor that preserves backgrounds while modifying non-rigid objects.
ProTakeaways
- CPAM is a zero-shot image editing framework from University of Science (VNU-HCM), Monash, and University of Dayton, accepted to IEEE TMM 2026.
- It rewrites self-attention and cross-attention to edit objects while keeping backgrounds pixel-stable, no fine-tuning needed.
- Works out of the box with Stable Diffusion 1.5, 2.1, and SDXL through mask-guided attention manipulation.
- New IMBA benchmark with 104 samples for object retention, modification, and background alteration.
- CPAM-SDXL cuts background LPIPS to 0.118 versus 0.182 for the best prior baseline, DiffEdit.
- Code and dataset are open-source; Gradio demo announced as coming soon.
A joint team from the University of Science (VNU-HCM), Monash University, and the University of Dayton has released CPAM, a zero-shot framework for one of the messiest problems in diffusion-based image editing: changing a specific object in a real photo without warping the rest of the scene. The work, accepted to IEEE Transactions on Multimedia, plugs into Stable Diffusion 1.5, 2.1, and SDXL with no fine-tuning.
When the background refuses to sit still
Text-guided editors like SDEdit, MasaCtrl, and Imagic tend to fail in predictable ways. They smear textures, lose object identity, demand extensive fine-tuning, and struggle to confine edits to a specific region. Ask them to change a dog's pose or swap a cup for a mug, and the wallpaper shifts, the lighting drifts, or the subject melts into something new.
The root cause lives in how diffusion models handle attention. Self-attention leaks features across the whole image, so preserving a background means fighting the model. Cross-attention lets a target prompt bleed into pixels it was never meant to touch. CPAM (Context-Preserving Adaptive Manipulation) attacks both failure modes directly.