Skip to main content
Back

DoReMi Beats Sonata by Routing 3D Scene Tokens on Geometry

A new Mixture-of-Experts design routes 3D point cloud data by geometric topology instead of semantics, hitting 80.1% mIoU on ScanNet and 77.2% on S3DIS.

Pro

Paper, Full Access
  • DoReMi proposes a topology-aware Mixture-of-Experts for unified 3D scene understanding across sensors.
  • Existing 3D MoE methods route by semantics and drop up to 19.8% mIoU under density changes.
  • Achieves 80.1% mIoU on ScanNet Val and 77.2% on S3DIS Area 5.
  • Uses frozen self-supervised Re branch plus Do branch with spatial-guided routing and entropy-based expert count.
  • Zero-shot on Matterport3D reaches 49.4% mIoU, beating Sonata and Point-MoE by clear margins.
  • Runs at 4.9 FPS on A100 with 147.5M active parameters; code release pending.

Training a single model to understand 3D scenes from every kind of sensor (LiDAR, RGB-D cameras, mesh reconstructions) is one of the messier problems in computer vision. The same wall looks like a dense triangle mesh in one dataset, a sparse ray-scan in another, and a partially occluded blob in a third. A new paper introduces DoReMi, a Mixture-of-Experts framework from Mingwei Xing, Xinliang Wang, and Yifeng Shi that argues the fix is to route point cloud tokens by their local geometry rather than by what object they belong to.

The paper reports 80.1% mIoU on the ScanNet validation set and 77.2% mIoU on S3DIS, edging out recent baselines like Sonata, PPT, and Point Transformer v3.

The routing bias nobody was talking about

Mixture-of-Experts has become the go-to trick for training one model across many domains: a gating network picks a handful of specialist sub-networks for each input token. In language models this works because tokens carry clear semantic identities. In 3D, the authors argue, blindly copying that recipe backfires.

Existing 3D MoE methods rely on rigid, semantic-driven routing. When an identical object such as a bed or floor undergoes topological changes such as density or completeness variations, vanilla MoE assigns the exact same experts, suffering performance drops of up to 19.8% mIoU under density variation. A "wall" gets sent to the wall expert regardless of whether it arrived as 200 dense mesh points or 20 sparse LiDAR returns. The domain gap in 3D lives in the geometry, and semantic routing is blind to it.