arXiv:2406.18544v1 Announce Type: new Abstract: 3D Gaussian Splatting (3DGS) has shown a powerful capability for novel view synthesis due to its detailed expressive ability and highly efficient rendering speed. Unfortunately, creating relightable 3D assets with 3DGS is still problematic, particularly for reflective objects, as its discontinuous representation raises difficulties in constraining geometries. Inspired by previous works, the signed distance field (SDF) can serve as an effective way for geometry regularization. However, a direct incorporation between Gaussians and SDF significantly slows training. To this end, we propose GS-ROR for reflective objects relighting with 3DGS aided by SDF priors. At the core of our method is the mutual supervision of the depth and normal between deferred Gaussians and SDF, which avoids the expensive volume rendering of SDF. Thanks to this mutual supervision, the learned deferred Gaussians are well-constrained with a minimal time cost. As the Gaussians are rendered in a deferred shading mode, while the alpha-blended Gaussians are smooth, individual Gaussians may still be outliers, yielding floater artifacts. Therefore, we further introduce an SDF-aware pruning strategy to remove Gaussian outliers, which are located distant from the surface defined by SDF, avoiding the floater issue. Consequently, our method outperforms the existing Gaussian-based inverse rendering methods in terms of relighting quality. Our method also exhibits competitive relighting quality compared to NeRF-based methods with at most 25% of training time and allows rendering at 200+ frames per second on an RTX4090.
The article “GS-ROR: Reflective Object Relighting with 3D Gaussian Splatting and Signed Distance Field Priors” introduces a novel approach for creating relightable 3D assets using 3D Gaussian Splatting (3DGS) and signed distance field (SDF) priors. While 3DGS has proven effective for view synthesis, it struggles with reflective objects due to its discontinuous representation. By incorporating SDF as a geometry regularization technique, the authors aim to overcome this limitation. However, directly combining Gaussians and SDF results in slower training. To address this, they propose GS-ROR, a method that achieves mutual supervision of depth and normal between deferred Gaussians and SDF, avoiding the need for expensive volume rendering. Additionally, they introduce an SDF-aware pruning strategy to remove Gaussian outliers, reducing floater artifacts. The results show that GS-ROR outperforms existing Gaussian-based inverse rendering methods in terms of relighting quality and is competitive with NeRF-based methods, while significantly reducing training time and allowing for real-time rendering.
Exploring GS-ROR: Enhancing Reflective Object Relighting with 3D Gaussian Splatting and SDF Priors
Relighting 3D assets, especially reflective objects, has always been a challenging task. While 3D Gaussian Splatting (3DGS) has shown promise in novel view synthesis with its detailed expressive ability and efficient rendering speed, creating relightable 3D assets using this method still poses difficulties due to its discontinuous representation and geometric constraints.
Inspired by previous works, researchers have found that incorporating signed distance fields (SDF) can be an effective way to regularize geometry in 3DGS. However, directly combining Gaussians and SDF in the training process can significantly slow down the training time.
To address this issue, a new approach called GS-ROR (Gaussian Splatting with SDF-aided Reflective Object Relighting) has been proposed. The core idea behind GS-ROR is the mutual supervision of depth and normal information between deferred Gaussians and SDF. This mutual supervision eliminates the need for expensive volume rendering of SDF, resulting in a minimal time cost for training.
By rendering the Gaussians in a deferred shading mode, the alpha-blended Gaussians maintain smoothness. However, individual Gaussians may still act as outliers, leading to floater artifacts. To overcome this problem, an SDF-aware pruning strategy has been introduced in GS-ROR. This strategy identifies and removes Gaussian outliers that are located far from the surface defined by SDF, effectively eliminating floater artifacts.
As a result of these innovations, GS-ROR outperforms existing Gaussian-based inverse rendering methods in terms of relighting quality for reflective objects. Furthermore, it achieves competitive relighting quality compared to NeRF-based methods but with significantly reduced training time, requiring only 25% of the training duration. Additionally, GS-ROR enables real-time rendering with a frame rate of 200+ frames per second on a high-end graphics card like RTX4090.
In conclusion, GS-ROR introduces a novel and efficient approach for relighting reflective objects using 3D Gaussian Splatting. By incorporating SDF priors and implementing mutual supervision, it overcomes the limitations of the traditional method, resulting in improved relighting quality. With its ability to achieve real-time rendering and reduced training time, GS-ROR opens up new possibilities for relighting in the field of computer graphics.
The paper titled “GS-ROR: Reflective Objects Relighting with 3D Gaussian Splatting aided by SDF Priors” introduces a novel approach to address the challenges of creating relightable 3D assets, particularly for reflective objects, using 3D Gaussian Splatting (3DGS) and signed distance field (SDF) priors.
The authors acknowledge that while 3DGS has demonstrated its effectiveness in novel view synthesis with its detailed expressive ability and efficient rendering speed, it falls short when it comes to handling reflective objects due to its discontinuous representation. This raises difficulties in constraining geometries and achieving accurate relighting.
To overcome these limitations, the authors propose GS-ROR, a method that combines 3DGS with SDF priors. By incorporating the SDF as a regularization technique for geometry, the authors aim to improve the accuracy and efficiency of relighting for reflective objects.
One of the key contributions of GS-ROR is the mutual supervision of depth and normal between deferred Gaussians and SDF. This approach avoids the computationally expensive volume rendering of SDF while ensuring that the learned deferred Gaussians are well-constrained. This mutual supervision helps achieve accurate relighting with minimal time cost.
Additionally, the paper introduces an SDF-aware pruning strategy to address the issue of Gaussian outliers. While the alpha-blended Gaussians in deferred shading mode generally produce smooth results, individual Gaussians may still be outliers, causing floater artifacts. The SDF-aware pruning strategy effectively removes these outliers located far from the surface defined by SDF, thus mitigating the floater issue.
According to the paper, the proposed GS-ROR method outperforms existing Gaussian-based inverse rendering methods in terms of relighting quality. It also exhibits competitive relighting quality compared to NeRF-based methods while requiring significantly less training time, with a rendering speed of 200+ frames per second on an RTX4090.
Overall, this paper presents an innovative approach to address the challenges of relighting reflective objects using 3DGS and SDF priors. The mutual supervision of depth and normal, along with the SDF-aware pruning strategy, contributes to improved relighting quality and efficiency. The results demonstrate the potential of GS-ROR as a valuable tool for relighting in various applications, such as virtual reality, gaming, and computer graphics.
Read the original article