HiddenObjects Crushes 72B Vision Models at Placing Objects in Scenes
A team turns diffusion inpainters into object-placement graders, distills the results into a 3.77 ms model that beats human annotations and large VLMs.
ProTakeaways
- HiddenObjects distills spatial priors for object placement from diffusion inpainters, producing 27M ranked box annotations across 27k scenes.
- Instead of removing objects, the pipeline inserts candidates and verifies with Grounded-SAM-2 plus ImageReward.
- Priors beat OPA human annotations on downstream image editing: 3.90 vs 2.68 on ImgEdit-Judge.
- Distilled DETR-style model hits 56.6% mAP, versus 1-3% for zero-shot Qwen2.5-VL-72B and OPA-trained baselines.
- Inference runs at 3.77 ms/image, a 230,000x speedup over the full extraction pipeline.
- Project page and heatmap visualizations available at hidden-objects.github.io.
Where should a pizza go in a photo of a kitchen? Where does a cat belong in a living room? These questions sound trivial, but they are surprisingly hard for machine learning systems that need to composite objects into scenes for image editing, data augmentation, or synthetic training data. A new paper from researchers at the Technical University of Denmark, the University of Copenhagen, and the Pioneer Centre for AI attacks this problem by extracting spatial priors directly from text-to-image diffusion models. It produces both a massive dataset and a distilled model that runs five orders of magnitude faster than the extraction pipeline.
The work introduces HiddenObjects, a dataset of 27 million ranked bounding-box placements across 27,000 real background scenes. The paper shows that priors learned from these annotations outperform sparse human labels by a wide margin on downstream image editing (3.90 vs 2.68 on the ImgEdit-Judge score) and crush zero-shot vision-language models on placement benchmarks.
The dirty secret of object placement datasets
Existing datasets for object placement come in two flavors, and both have problems. The manual route (OPA, DreamEditBench, MureCom) gives you clean backgrounds but only a handful of labeled locations per instance, and the labels are noisy because humans disagree on where objects should go.
The automated route (PIPE, SAM-FB) is the more insidious problem. These pipelines start with a real image containing an object, then remove that object via inpainting to create a training pair. Removal is effective at deleting the requested content but leaves visible traces that facilitate shortcut learning: blurry patches, residual shadows, and larger artifacts. A model trained on this data learns to detect the ghost of the erased object rather than reason about scene semantics.