IAFS Fixes AI Image Upscaling's Biggest Trade-off Without Any Retraining
A training-free inference-time technique for diffusion super-resolution that separates images into frequency bands and iteratively refines them to beat the perception-fidelity trade-off.
ProTakeaways
- IAFS is a training-free inference-time method for diffusion super-resolution that beats the perception-fidelity trade-off.
- It splits each particle into high/low-frequency parts and fuses low-frequency info from similar neighbors via AdaIN.
- Different reward mixes (CLIPIQA and LPIPS) are used at different timesteps based on frequency behavior.
- Iterative refinement uses each round's output as pseudo-GT, with n=3 iterations offering the best balance.
- Tested on ResShift, DiffBIR, and SeeSR across ImageNet, DIV2K, RealSR, DRealSR.
- Paper reports gains on both structural (PSNR/SSIM) and perceptual (LPIPS/CLIPIQA/MUSIQ/MANIQA) metrics simultaneously.
Diffusion-based super-resolution has a well-known split personality. Push it toward crisp textures and edges look invented. Push it toward faithful reconstruction and everything turns plasticky. A new training-free method called Iterative Inference-Time Scaling with Adaptive Frequency Steering (IAFS) attacks this trade-off by treating the problem as one of frequency allocation, and it improves both sides of the ledger at once across three popular diffusion SR backbones.
Existing inference-time scaling tricks each pick a poison. Particle optimization methods often produce poor perceptual quality, while optimal-path search methods lose structural fidelity. IAFS is designed to sit between them.

Frequencies beat rewards as the right handle
The authors work out something clean in the Fourier domain: because noise variance in the reverse process shrinks over time, the amplitude spectrum of the intermediate latent decays as a Gaussian in frequency magnitude. In practice, low-frequency structure is largely fixed early in the reverse diffusion, while high-frequency detail is only recovered in the final steps. They also lean on a useful complementarity between two common metrics: CLIPIQA is sensitive to high-frequency content, while LPIPS favors mid-to-low-frequency structure.