Solving image-to-3D from a single view has traditionally been a challenging problem, with existing neural reconstruction methods relying on scene-specific optimization. However, these methods often struggle with generalization and consistency. To address these limitations, we introduce a novel neural rendering technique called Hyper-VolTran.
Unlike previous approaches, Hyper-VolTran employs the signed distance function (SDF) as the surface representation, allowing for greater generalizability. Our method incorporates generalizable priors through the use of geometry-encoding volumes and HyperNetworks.
To generate the neural encoding volumes, we utilize multiple generated views as inputs, enabling flexible adaptation to novel scenes at test-time. This adaptation is achieved through the adjustment of SDF network weights conditioned on the input image.
In order to improve the aggregation of image features and mitigate artifacts from synthesized views, our method utilizes a volume transformer module. Instead of processing each viewpoint separately, this module enhances the aggregation process for more accurate and consistent results.
By utilizing Hyper-VolTran, we are able to avoid the limitations of scene-specific optimization and maintain consistency across images generated from multiple viewpoints. Our experiments demonstrate the advantages of our approach, showing consistent results and rapid generation of 3D models from single images.
Abstract:Solving image-to-3D from a single view is an ill-posed problem, and current neural reconstruction methods addressing it through diffusion models still rely on scene-specific optimization, constraining their generalization capability. To overcome the limitations of existing approaches regarding generalization and consistency, we introduce a novel neural rendering technique. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Specifically, our method builds neural encoding volumes from generated multi-view inputs. We adjust the weights of the SDF network conditioned on an input image at test-time to allow model adaptation to novel scenes in a feed-forward manner via HyperNetworks. To mitigate artifacts derived from the synthesized views, we propose the use of a volume transformer module to improve the aggregation of image features instead of processing each viewpoint separately. Through our proposed method, dubbed as Hyper-VolTran, we avoid the bottleneck of scene-specific optimization and maintain consistency across the images generated from multiple viewpoints. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.