ETH Zürich's ZipSplat Breaks 3D Scene Reconstruction With 2dB Quality Gains
ZipSplat uses k-means clustering to compress scene tokens, achieving state-of-the-art 3D reconstruction with 6x fewer Gaussians and no camera poses required.
PRO- ZipSplat reconstructs 3D scenes from unposed images in one forward pass, using ~6x fewer Gaussians than pixel-aligned methods while achieving state-of-the-art quality.
- Instead of one Gaussian per pixel, it compresses visual tokens via k-means clustering into a compact set of scene tokens, then decodes each into a group of 3D Gaussians.
- A single compression parameter (0 to 1) controls Gaussian count at inference -- no retraining needed to trade quality for speed.
- Sets new SOTA on DL3DV (+2.1 dB) and RealEstate10K (+1.2 dB) over the best pose-free baseline; generalizes zero-shot to Mip-NeRF360 and ScanNet++.
- No camera poses or intrinsics required; optional pose priors further boost PSNR (RE10K: 26.20 → 27.19 with priors).
- Code, training framework, and pretrained weights (CC BY-NC 4.0) are open-sourced on GitHub; weights auto-download from Hugging Face.
Every pixel gets a Gaussian. That has been the unspoken rule of feed-forward 3D Gaussian Splatting since the field took off. It is a simple, differentiable, and surprisingly effective strategy -- but it has a fundamental flaw: it ties the reconstruction budget to camera resolution, not to what is actually in the scene. A blank wall and a detailed sculpture get the same number of Gaussians. ZipSplat, from researchers at ETH Zürich and Microsoft, breaks that rule entirely.
The pixel-alignment trap
To understand why this matters, it helps to know how current feed-forward 3DGS methods work. Existing methods predominantly rely on a pixel-aligned Gaussian prediction paradigm, where each 2D pixel is mapped to a 3D Gaussian. The appeal is obvious: you get a dense, differentiable mapping from image to 3D with no extra machinery.
But the problems compound quickly. The 3D Gaussians are tightly coupled to the 2D pixels, resulting in view-biased Gaussian densities -- regions frequently observed across views generate more Gaussians, whereas texture-rich or structurally complex areas may be underrepresented. Feature extraction and cross-view interactions occur in 2D image space, limiting interactions among neighboring points in 3D space and causing floating artifacts. Geometric consistency is difficult to guarantee, and redundant Gaussians often lead to blur.
The result is a representation that wastes compute on flat, featureless regions while potentially under-serving complex geometry. And because the Gaussian count is fixed by resolution, you cannot trade quality for speed without retraining the whole model.
Zip it: compressing the scene into tokens
ZipSplat is a feed-forward model for 3D Gaussian Splatting that reconstructs a scene from a few unposed images in a single forward pass, with no poses, intrinsics, or per-scene optimization. Instead of emitting one Gaussian per pixel, it compresses the input views into a compact set of scene tokens and decodes each into a small group of Gaussians, reaching state-of-the-art quality with far fewer Gaussians and a single knob to trade quality for size.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
