SGS-SLAM Bakes Semantic Labels Directly Into 3D Gaussian Maps
A trending open-source SLAM system fuses 3D Gaussian Splatting with semantic segmentation to deliver object-aware dense reconstruction in real time.
- SGS-SLAM is the first semantic visual SLAM built on 3D Gaussian Splatting, released under BSD-3 on GitHub.
- Each Gaussian jointly stores appearance, geometry and a semantic class, optimized through a multi-channel loss.
- A semantic feature loss and semantic-guided keyframe selection fix oversmoothing and drift in object-level geometry.
- Reports state-of-the-art pose, reconstruction and segmentation on Replica, ScanNet and ScanNet++ with real-time rendering.
- Published at ECCV 2024; arXiv paper already has 95+ citations and multiple follow-ups.
- RGB-D only, requires ground-truth semantic masks, and targets bounded indoor scenes.
SGS-SLAM embeds semantic labels in 3D Gaussian maps
Mingrui Li, Shuhong Liu and collaborators have published the complete SGS-SLAM repository, including tracking, mapping, post-optimization and visualization code under the BSD-3-Clause license. Presented at ECCV 2024, the system is described by its authors as the first semantic visual SLAM pipeline built on 3D Gaussian Splatting.
Visual simultaneous localization and mapping estimates a camera’s trajectory while reconstructing its surroundings. SGS-SLAM represents that reconstruction as a collection of Gaussian primitives, each defined by properties such as position, shape, opacity, color and semantic features. Those explicit primitives can be rendered quickly and updated individually as new RGB-D frames arrive.
Semantics become map data
Dense SLAM systems based on neural implicit fields, including NICE-SLAM, ESLAM and Point-SLAM, can blur object boundaries and lose thin or weakly textured geometry. SGS-SLAM jointly optimizes appearance, geometry and semantic features within the Gaussian map, preserving class information in the same representation used for reconstruction and rendering.
- Joint channel optimization. Each Gaussian carries appearance, geometric and semantic parameters that are updated during tracking and mapping.
- Semantic feature loss. Semantic supervision supplies an additional optimization signal where color and depth provide limited evidence. The paper reports better reconstruction of thin and low-texture objects as a result.
- Semantic keyframe selection.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
