NaviNeRF++: Towards Interpretable 3D Reconstruction via Unsupervised Disentangled Representation Learning.

Abstract

3D reconstruction is a pivotal technology that recreates three-dimensional structures from two-dimensional representations, facilitating AI's understanding and interaction with the real world. However, existing methods pose challenges from two perspectives, i.e., a lack of understanding of the semantics behind 3D representations and the excessive reliance on extra priors to achieve 3D control. To address these challenges, we propose an interpretable 3D reconstruction framework, dubbed NaviNeRF++, to discover and identify the underlying 3D semantics by integrating multimodal large language models (MLLMs) and Neural Radiance Fields (NeRF). The model achieves fine-grained 3D disentanglement from the perspective of disentangled representation learning (DRL), while preserving high-quality and view-consistent 3D reconstruction. Specifically, the framework consists of three key components: i) a lightweight 2D perception module designed to derive an orthogonal and well-disentangled latent space with knowledge distilled from a pre-trained DRL model; ii) a NeRF-based 3D navigation module dedicated to finding semantic factors in the learned latent space, while concurrently enabling high-quality 3D reconstruction; and iii) an attribute identification module that identifies textual concepts of semantic factors by leveraging the commonsense knowledge of MLLMs. To our knowledge, this work is the first to achieve interpretable 3D reconstruction and fine-grained 3D disentanglement in an unsupervised manner. Empirical results further demonstrate its superior performance compared to off-the-shelf solutions.