UC San Diego

Extended Reality (XR) encompasses immersive technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), which blend the physical and digital worlds. For XR experiences to captivate users and seamlessly integrate with reality, photorealistic content is essential. Photorealism ensures that virtual elements convincingly interact with real-world environments, enhancing immersion and fostering a sense of presence for users. This dissertation explores various methods to facilitate the convenient and efficient creation of photorealistic digital content from images for diverse subjects.

Creating photorealistic content from images involves estimating intrinsic components from scenes, a highly challenging and ill-posed problem. To ensure photorealism, this dissertation focuses on synthesizing spatially-varying Bidirectional Reflectance Distribution Functions (BRDFs) or textures, modeling complex light transport, and leveraging large-scale real-world data. Firstly, we discuss how existing priors can be used for material and lighting transfer from images to 3D scene geometry. Secondly, we explore the utilization of diffusion models pre-trained on large-scale real-world images as priors for high-quality texture synthesis and transfer to arbitrary 3D shapes with image inputs. Additionally, specialized objects like transparent shapes or portraits are addressed through learning-based approaches with synthetic data and synthetic-to-real adaptation for complex light transport and relighting to handle specific appearances.

The key contribution of this dissertation is developing efficient methods to create high-quality photorealistic content for XR with minimal human effort. Unlike prior works that depend on elaborate capture systems or extensive image sets, this dissertation achieves photorealism using just a few images easily captured from commercial mobile devices. We demonstrate diverse, high-quality photorealistic content produced by our methods, suitable for various XR applications.