HKU's Pair2Scene Generates Dense 14-Object Rooms Without Bigger Training Data
A new framework generates 3D indoor scenes by learning pairwise object rules, producing layouts three times denser than training data while staying physically plausible.
ProTakeaways
- Pair2Scene generates 3D indoor scenes by learning pairwise local object relations instead of global joint distributions.
- Two relation types are modeled: physical support hierarchies and semantic functional proximity between objects.
- A Point-MAE plus Transformer network predicts a Mixture of Logistics over the dependent object's bounding box.
- New 3D-Pairs dataset aggregates 138,429 relational tuples from 3D-Front, MesaTask, and InternScenes.
- Beats ATISS and DiffuScene on FID and KID while scaling from 4 to 14 objects per room.
- Runs in 69 seconds per dense scene, roughly 5x faster than LayoutVLM.
Generating realistic 3D indoor scenes has been stuck in a frustrating loop. Neural models trained on scene datasets can only produce layouts as complex as what they saw during training, and language model based approaches sound smart but fumble the actual geometry. A team from HKU, Tsinghua, and SJTU is proposing a different bet with Pair2Scene: stop trying to model the whole room at once, and instead learn the local rules that govern how any two objects sit next to each other.
The paper, accepted at ICML 2026, argues that object placement relies mainly on local dependencies rather than information-redundant global distributions. A lamp cares about the nightstand under it, not about the couch across the room. Model those pairwise dependencies well, and you can procedurally assemble arbitrarily complex scenes.
Why global scene models keep hitting a wall
Two families dominate 3D layout generation today, and both have structural problems. Learning based methods like ATISS and DiffuScene fit the joint distribution of an entire scene, which caps output complexity at the training set. On the curated 3D-Front bedroom split used in this paper, the average room contains only 4.07 objects, so generated layouts look sparse. LLM and VLM guided methods like Holodeck or LayoutVLM inject commonsense, but current approaches often struggle to scale beyond training distribution to dense scenes or lean on models that lack precise spatial reasoning, producing floating chairs and interpenetrating furniture.