UC San Diego

Digital 3D content plays an important role in many applications such as video games, animations, virtual reality (VR) and augmented reality (AR). Traditionally, the creation of digital 3D content requires complicated software and designers with special expertise, which imposes great challenges for novice users. In comparison, an alternative approach for this task is to acquire and digitize the appearance of real-world scenes by capturing images and automatically reconstructing 3D models from the scans. In this dissertation, we present a series of works for efficiently and accurately creating 3D representations from the captured images. We exploit appropriate representations for both scene geometry and reflectance to support different functionalities including novel view synthesis, relighting and dynamic animations.

First, we present an approach for generating accurate texture maps for RGB-D reconstructions that enable us to navigate the scene under novel viewpoints. Our method can correct misalignments between captured images caused by inaccurate camera poses and corrupted geometries and produce highly-quality texture maps. Afterwards we take one step further and propose a learning-based method to reconstruct high-quality meshes with per-vertex BRDFs from a sparse set of six images captured under collocated camera and light, which supports visualization of the scene under novel viewpoints and lighting conditions. Then we go beyond traditional mesh representations and propose to learn a novel volumetric representation that encodes volume density, normal and reflectance properties at any arbitrary 3D point in the scene for joint view synthesis and relighting. We demonstrate that our volumetric representation can be estimated from images captured with a simple collocated camera-light setup, and accurately model the appearance of real-world scenes with complex geometry and reflectance. Finally, we develop approaches for modeling the dynamic appearance of human faces and learning animatable lifelike avatars that support free-viewpoint relighting and novel expressions. We apply neural networks to directly regress the facial geometry and textures under the desired viewpoints, lightings and expressions. We show that our model can be animated and driven by images captured with VR-headset mounted cameras, demonstrating the first system for face-driven interactions in VR that uses a photorealistic relightable face model.