SplatSDF: Boosting Neural Implicit SDF via Gaussian Splatting Fusion

Runfa Blark Li*†, Keito Suzuki*, Bang Du*, Ki Myung Brian Lee†, Nikolay Atanasov†, Truong Nguyen*
*Video Processing Lab & †Existential Robotics Lab, UC San Diego
Paper arXiv Code

Abstract

A signed distance function (SDF) is a useful representation for continuous-space geometry and many related operations, including rendering, collision checking, and mesh generation. Hence, reconstructing SDF from image observations accurately and efficiently is a fundamental problem. Recently, neural implicit SDF (SDF-NeRF) techniques, trained using volumetric rendering, have gained a lot of attention. Compared to earlier truncated SDF (TSDF) fusion algorithms that rely on depth maps and voxelize continuous space, SDF-NeRF enables continuous-space SDF reconstruction with better geometric and photometric accuracy. However, the accuracy and convergence speed of scene-level SDF reconstruction require further improvements for many applications. With the advent of 3D Gaussian Splatting (3DGS) as an explicit representation with excellent rendering quality and speed, several works have focused on improving SDF-NeRF by introducing consistency losses on depth and surface normals between 3DGS and SDF-NeRF. However, loss-level connections alone lead to incremental improvements. We propose a novel neural implicit SDF called "SplatSDF" to fuse 3DGS and SDF-NeRF at an architecture level with significant boosts to geometric and photometric accuracy and convergence speed. Our SplatSDF relies on 3DGS as input only during training, and keeps the same complexity and efficiency as the original SDF-NeRF during inference. Our method outperforms state-of-the-art SDF-NeRF models on geometric and photometric evaluation by the time of submission.

Overview

We focus on three modules - Dynamic GS (a) Segmentation & Flow, (b) Management and (c) Tracking & Prediction. DynaGSLAM takes RGBD sequence as input to construct map with GS, (a) segment dynamic GS from static GS in 3D, and estimate dynamic GS 3D motion flow between frames. (b) Dynamic GS are managed separately from static GS with GS flow, but combined to jointly optimize. Case 1&2 are the rules for dynamic GS adding; "Cond. 1&2" denotes the conditions for dynamic GS deletion. (c) The optimized dynamic GS at current and past frames are used to interpolate/extrapolate dynamic GS in the continuous timeline from past to future. "CHS" refers to "cubic Hermite spline" and "LF" refers to "linear function". Please refer to the paper for details.

Results

Scene

Epochs

Neuralangelo SplatSDF

Epochs

3D Model Comparison

Neuralangelo

Ours

BibTeX

@misc{splatsdf,
      title={SplatSDF: Boosting Neural Implicit SDF via Gaussian Splatting Fusion}, 
      author={Runfa Blark Li and Keito Suzuki and Bang Du and Ki Myung Brian Lee and Nikolay Atanasov and Truong Nguyen},
      year={2024},
      eprint={2411.15468},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.15468}, 
}