UC Santa Cruz

Three-dimensional (3D) reconstruction, a popular topic in computer vision, has been researched extensively for more than three decades. Many practitioners have proposed several image-based Structure-from-Motion (SfM) and visual Simultaneous Localization and Mapping (SLAM) algorithms to improve the quality, accuracy, and efficiency of 3D reconstruction results. Nevertheless, the 3D reconstruction of human-made indoor structures remains one of the most challenging problems since indoor environments present specific challenges due to their distinctive properties such as lack of textures and dramatic viewpoint changes.

This dissertation proposes a novel end-to-end approach for the 3D reconstruction of indoor scenes under the ``Manhattan World'' (MW) constraint, which assumes that visible planes intersect at orthogonal angles. My algorithm recovers planar structures by first computing multi-planar segmentation and motion estimation from consecutive image pairs using MW-constrained homographies. It then proceeds to recover the relative scale at each frame with respect to the global reference system to determine chains of clusters, where each cluster is associated with a plane in the scene. Camera motion and scene planar geometry are then optimized using a novel formulation of Bundle Adjustment. Compared with other state-of-the-art SfM/SLAM algorithms that maintain a representation of individual 3D points in space as in traditional reconstruction techniques, my algorithm estimates 3D plane locations that contain observed points. Using the estimated plane information, the algorithm renders input images using estimated 3D planes in a planar-patch-wise fashion. Thanks to the decreased number of parameters, this enables the generation of dense reconstructions with fewer images and facilitates real-world mobile applications.