4D monocular surgical reconstruction under arbitrary camera motions.

Abstract

Reconstructing deformable surgical scenes from endoscopic videos is a challenging task with important clinical applications. Recent state-of-the-art approaches, such as those based on implicit neural representations or 3D Gaussian splatting, have made notable progress in this area. However, most existing methods are designed for deformable scenes with fixed endoscope viewpoints and rely on stereo depth priors or accurate structure-from-motion for both initialization and optimization. This limits their ability to handle monocular sequences with large camera movements, restricting their use in real clinical settings. To address these limitations, we propose Local-EndoGS, a high-quality 4D reconstruction framework for monocular endoscopic sequences with arbitrary camera motion. Local-EndoGS introduces a progressive, window-based global scene representation that allocates local deformable scene representations for each observed window, enabling scalability to long sequences with substantial camera movement. To overcome unreliable initialization due to the lack of stereo depth or accurate structure-from-motion, we propose a coarse-to-fine initialization strategy that integrates multi-view geometry, cross-window information, and monocular depth priors, providing a robust foundation for subsequent optimization. In addition, we incorporate long-range 2D pixel trajectory constraints and physical motion priors to improve the physical plausibility of the recovered deformations. We comprehensively evaluate Local-EndoGS on three public endoscopic datasets with deformable scenes and varying camera motions. Local-EndoGS achieves superior performance in both appearance quality and geometry, consistently outperforming state-of-the-art methods. Extensive ablation studies further validate the effectiveness of our key designs. Our code will be released upon acceptance at https://github.com/IRMVLab/Local-EndoGS.