HandNeRF++: Modeling Animatable Interacting Hands With Neural Radiance Fields.

Abstract

In this work, we explore the rendering of photo-realistic free-viewpoint hand pose animation. We present HandNeRF, the first NeRF-based framework to reconstruct accurate appearance and geometry for interacting hands. To overcome the texture contamination and shape artifact problems when dealing with complex interacting scenarios, we further introduce HandNeRF++ to achieve better performance. In our advanced framework, a pose-driven deformation field is designed to establish correspondence from diverse poses to a canonical space, where the pose- and shape-disentangled NeRFs are optimized. To enhance the geometry and texture cues in rarely-observed areas for interacting hands, we establish a connection between the interacting hands by proposing the adaptive hand-sharing technique for cross-hand augmentation. Meanwhile, we further leverage the hand poses to generate fine-grained density priors, serving as valuable guidance for occlusion-aware geometry learning. Furthermore, a neural feature distillation method and a neural refiner are proposed to facilitate color optimization and further polish the renderings. With the collaboration of all the modules and strategies, our HandNeRF++ significantly advances the capabilities of NeRF-based 3D reconstruction in the context of interacting hands. Extensive experiments are conducted to validate the merits of the proposed frameworks. We report a series of state-of-the-art results both qualitatively and quantitatively.