UC Irvine

This dissertation explores the challenges and solutions in physically-based inverse rendering, aiming to reconstruct detailed 3D scenes—including geometry, material properties, and lighting—from ordinary 2D images. The research journey began with the goal of making differentiable rendering more broadly applicable, particularly in scenarios where existing methods struggled with complex shapes, difficult initializations, or intricate material and lighting effects.

The first step involved bringing differentiability to time-gated rendering, which is essential for advanced tasks such as Non-Line-of-Sight (NLOS) reconstruction that rely on the timing of light as it moves through a scene. This work resulted in a new theoretical framework that extends path-space differentiation to handle path-length importance, introducing a key boundary term needed for accurate gradients in time-resolved problems. This exploration also highlighted the need for more flexible and robust geometric representations, even in conventional inverse rendering settings.

To address these needs, a new inverse rendering pipeline was developed that combines the strengths of both implicit and explicit geometric representations. By leveraging differentiable iso-surface extraction, this approach allows for dynamic changes in shape during optimization, while retaining the accuracy of mesh-based rendering for effects such as soft shadows and interreflections.

Despite these advances, obtaining a good initial estimate of the scene remained a challenge. To accelerate this process, neural rendering techniques were incorporated, drawing on recent progress in neural volumetric surface reconstruction (such as NeRF and NeuS). These methods provide strong initial estimates for geometry, materials, and lighting, which are then refined using a physics-based differentiable renderer capable of handling complex global illumination. This hybrid approach combines the resilience of neural methods with the precision of physical simulation.

Standard neural methods, however, often struggle with highly glossy objects, and traditional lighting models—such as infinite-distance environment maps—fail to capture important parallax and near-field effects present in real indoor scenes. To address these challenges, EnvMAP++ was introduced as a lightweight, parallax-aware lighting model, and Neural Implicit Evolution (NIE) was integrated for robust, topology-adaptive shape optimization, even when the initial geometry is imperfect. Paired with efficient sampling strategies for stable gradient estimation of specular reflections, the resulting system faithfully captures the complex interplay between shape, glossy materials, and detailed local lighting.

Collectively, these projects form a cohesive set of tools and methods that make physically-based inverse rendering more accurate, flexible, and reliable, helping to advance the field of computational scene understanding from images.