Spherical Harmonic (SH) lighting is widely used for real-time rendering within Precomputed Radiance Transfer (PRT) systems. SH coefficients are precomputed and stored at object vertices, and combined interactively with SH lighting coefficients to enable effects like soft shadows, interreflections, and glossy reflection. However, the most common PRT techniques assume distant, low-frequency environment lighting, for which SH lighting coefficients can easily be computed once per frame. There is currently limited support for near-field illumination and area lights, since it is non-trivial to compute the SH coefficients for an area light, and the incident lighting (SH coefficients) varies over the object geometry. We present an efficient closed-form solution for projection of uniform polygonal area lights to spherical harmonic coefficients of arbitrary order, enabling easy adoption of accurate area lighting in PRT systems, with no modifications required to the core PRT framework. Our method only requires computing zonal harmonic (ZH) coefficients, for which we introduce a novel recurrence relation. In practice, ZH coefficients are built up iteratively, with computation linear in the desired SH order. General SH coefficients can then be obtained by the recently developed sparse zonal harmonic rotation method.

This thesis presents three systems for editing the motion of videos. First, selectively de-animating videos seeks to remove the large-scale motions of one or more objects so that other motions are easier to see. The user draws strokes to indicate the regions that should be immobilized, and our algorithm warps the video to remove large-scale motion in regions while leaving finer-scale, relative motions intact. We then use a graph-cut-based optimization to composite the warped video with still frames from the input video to remove unwanted background motion. Our technique enables applications such as clearer motion visualization, simpler creation of artistic cinemagraphs, and new ways to edit appearance and motion paths in video. Second, we design a fully automatic system to create portrait cinemagraphs by tracking facial features and de-animating the video with respect to the face and torso. We then generate compositing weights automatically to create the final cinemagraph portraits.

Third, we present a user-assisted video stabilization algorithm that is able to stabilize challenging videos when state-of-the-art automatic algorithms fail to generate a satisfactory result. Our system introduces two new modes of interaction that allow the user to improve an unsatisfactory automatically stabilized video. First, we cluster tracks and visualize them on the warped video. The user ensures that appropriate tracks are selected by clicking on track clusters to include or exclude them to guide the stabilization. Second, the user can directly specify how regions in the output video should look by drawing quadrilaterals to select and deform parts of the frame. Our algorithm then computes a stabilized video using the user-selected tracks, while respecting the user-modified regions.

Modeling the appearance of real scenes from captured images is one key problem in computer graphics and computer vision. This traditionally requires a large number of input samples (e.g. images, light-view directions, depth hypotheses, etc.) and consumes extensive computational resources. In this dissertation, we aim to make scene acquisition more efficient and practical, and we present several approaches that successfully reduce the required number of samples in various appearance acquisition problems.

We exploit techniques to explicitly reconstruct the geometry and materials in a real scene; the two components essentially determine the scene appearance. On the geometry side, we introduce a novel deep multi-view stereo technique that can reconstruct high-quality scene geometry from a sparse set of sampling depth hypotheses. We leverage uncertainty estimation in a multi-stage cascaded network, which reconstructs highly accurate and highly complete geometry with low costs in a coarse-to-fine framework. On the material side, the reflectance of a real material is traditionally measured by tens and even hundreds of captured images. We present a novel reflectance acquisition technique that can reconstruct high-fidelity real materials from only two near-field images.

Moreover, we exploit image-based acquisition techniques that bypass explicit scene reconstruction and focus on realistic image synthesis under new conditions. We first present a novel deep neural network for image-based relighting. Our network simultaneously learns optimized input lighting directions and a relighting function. Our approach can produce photo-realistic relighting results under novel environment maps from only five images captured under five optimized directional lights. We also study the problem of view synthesis for real objects under controlled lighting, which classically requires dense input views with small baselines. We propose a novel deep learning based view synthesis technique that can synthesize photo-realistic images from novel views across six widely-spaced input views. Our network leverages visibility-aware attention information to effectively aggregate multi-view appearance. We also show that our view synthesis technique can be combined with our relighting technique to achieve novel-view relighting from sparse light-view samples.

Complex materials and lighting conditions can produce remarkably high-quality renderings with photorealism. However, efficiently rendering them remains challenging due to the high complexity of physically based light transport simulation. To improve the efficiency of rendering complex scenes, one common approach is using prefiltering techniques to precompute simplified material or lighting models. But usually these prefiltered models cannot preserve their original appearance accurately, leading to undesired rendering results.

In this dissertation, we present a series of novel appearance preserving prefiltering techniques. Our goal is to compute prefiltered material and lighting models that have smaller storage sizes and enable efficient rendering of complex scenes. Images rendered using the prefiltered model should closely resemble the overall appearance given by the original model.

Our first work addresses the challenge of prefiltering volumetric scattering models. We present a joint optimization of multiple material scattering parameters, i.e., single-scattering albedos and phase functions, to accurately prefilter heterogeneous and anisotropic media. Our method leads to significantly better accuracy compared to traditional linear downsampling and offers several orders of magnitude storage reduction.

Our second work focuses on prefiltering the reflectance of a displacement-mapped surface. We represent the prefiltered surface reflectance model as a spatially varying bidirectional reflectance distribution function (SVBRDF) at a reduced resolution. To express our appearance preserving SVBRDF efficiently, we decompose it into a spatially varying normal distribution function (SVNDF) and a novel scaling function that accurately captures micro-scale changes of shadowing, masking, and interreflection effects. The scaling function can further be computed by an efficient factorization. Our prefiltering method generalizes well to different types of surfaces and enables anti-aliased level-of-detail rendering.

Finally, our last work turns to the challenge caused by complex lighting conditions. Given a scene with many polygonal area lights, we prefilter all the lights into a sparse 3D grid of spherical harmonic (SH) lighting coefficients and gradients, allowing interpolating SH coefficients accurately at any intermediate point. This enables scaling real-time precomputed radiance transfer (PRT) to hundreds of area lights. Our method builds on a novel analytic formula for SH gradients, which may benefit many other applications beyond rendering.

Most physical objects have a material associated with them: from the wooden surface of a table to a velvet upholstery of a couch. The materials play a crucial role in the appearance of objects. Materials define the interaction of a surface with the light. Without good material models, realism in rendering cannot be achieved. In this dissertation, we will discuss how by using neural networks we can compactly and efficiently represent a wide variety of materials. Not only can neural networks provide excellent compression ratios, but they can be easily integrated into existing rendering engines.Our first contribution explores how we can use deep learning to render specular appearances. Many materials are made of highly reflective surfaces: for example, brushed metal or silky fabrics. Traditionally, many methods would model materials explicitly. However, those approaches require high storage requirements and a lot of computation during the simulation. Instead, we propose to use a generative adversarial network (GAN) to learn a high-frequency appearance from real and synthetic materials. We also propose to use texture synthesis on a feature texture. Our second contribution, NeuMIP, allows us to represent a wide variety of materials at different scales using neural networks. Traditional MIPMAP methods fail to downsample complex materials correctly. They naively average parameters over a patch, resulting in an incorrect appearance. NeuMIP avoids this problem by optimizing the neural textures at different levels of detail independently. A Neural Offset Module enables support for geometrically complex materials. NeuMIP assumes the material is applied to an infinitely large surface. As a result, material appearance on real objects would significantly diverge from the reference at the grazing angles. In our third work, we propose to take curvature into account and train on cylinders with different radii. Because of that, we can generate the natural silhouette of the object. Also, we support semitransparent materials. Neural materials are a novel way to represent complex real and synthetic materials and can be easily integrated into existing rendering pipelines. They offer high-quality and fast evaluation. We hope neural-based methods will become a standard way to represent materials in the future.

Digital 3D content plays an important role in many applications such as video games, animations, virtual reality (VR) and augmented reality (AR). Traditionally, the creation of digital 3D content requires complicated software and designers with special expertise, which imposes great challenges for novice users. In comparison, an alternative approach for this task is to acquire and digitize the appearance of real-world scenes by capturing images and automatically reconstructing 3D models from the scans. In this dissertation, we present a series of works for efficiently and accurately creating 3D representations from the captured images. We exploit appropriate representations for both scene geometry and reflectance to support different functionalities including novel view synthesis, relighting and dynamic animations.

First, we present an approach for generating accurate texture maps for RGB-D reconstructions that enable us to navigate the scene under novel viewpoints. Our method can correct misalignments between captured images caused by inaccurate camera poses and corrupted geometries and produce highly-quality texture maps. Afterwards we take one step further and propose a learning-based method to reconstruct high-quality meshes with per-vertex BRDFs from a sparse set of six images captured under collocated camera and light, which supports visualization of the scene under novel viewpoints and lighting conditions. Then we go beyond traditional mesh representations and propose to learn a novel volumetric representation that encodes volume density, normal and reflectance properties at any arbitrary 3D point in the scene for joint view synthesis and relighting. We demonstrate that our volumetric representation can be estimated from images captured with a simple collocated camera-light setup, and accurately model the appearance of real-world scenes with complex geometry and reflectance. Finally, we develop approaches for modeling the dynamic appearance of human faces and learning animatable lifelike avatars that support free-viewpoint relighting and novel expressions. We apply neural networks to directly regress the facial geometry and textures under the desired viewpoints, lightings and expressions. We show that our model can be animated and driven by images captured with VR-headset mounted cameras, demonstrating the first system for face-driven interactions in VR that uses a photorealistic relightable face model.

Human portraits are ubiquitous in our everyday life. However, after we take the portraits using the camera, the lighting condition of the captured image cannot be easily changed. Previous works have investigated how to change the lighting condition of a portrait, but they usually require special hardware to achieve relighting. In this thesis, we study the problem of portrait relighting, aiming at accuracy and practicality. We present several novel approaches that enable relighting portraits after the capture.With the goal of practical relighting that can be used in everyday life, we first develop single image portrait relighting. We leverage the prior of human portraits, and train a neural network that learns to perform relighting given only a single portrait. Our network can achieve convincing relighting results with very limited computation power, enabling real-time portrait relighting applications. Next, we focus on how to achieve precise relighting under point lights. We capture the portrait using discrete point lights under a light stage, and super-sample the location of the point lights to achieve high-frequency relighting. This enables finer controls on the shadows on human faces. Finally, we investigate changing both the viewpoint and the lighting condition of captured portraits. To this end, we propose neural light-transport field (NeLF). Given only 5 frontal portraits, this neural rendering approach can generate portraits from novel views under arbitrary natural lightings.

Video stabilization is one of the most widely sought features in video processing. The problem of video stabilization typically consists of two steps: video motion analysis and stabilized frame synthesis. Traditional video motion analysis relies heavily on local image features and is prone to error in the presence of occlusions and motion blur. It is also challenging to avoid visual artifacts in the stabilized frame synthesis due to complexity of the scenes, e.g. parallax, dynamic objects. In this dissertation, we focus on exploring video stabilization methods that generate high quality stable results and are generalizable to real world videos.

We start from selfie videos, which is a special type of video that is challenging for all video stabilization methods due to the dynamic occlusion. Our solution is to analyze motion of the foreground/background separately and jointly stabilize the motion. Specifically, we seek to use 3D human face model as a prior information of foreground motion. For the background, our approach tracks random pixels using optical flow. We exploit non-linear least squares optimization to stablize both the foreground/background motion. To make the process practical for commercial applications, we also designed an online version of the pipeline using a sliding window scheme. We also exploit deep learning techniques to replace optimization, resulting in orders of magnitude speed improvement compared to the state-of-the-art optimization based approaches.

Our works also generalize to the general video stabilization context. General video occlusion is free-form, therefore a more general scheme is required to appropriately constrain the video stabilization. We proposed two different solutions based on this principle. Both of our approaches use optical flow to analyze the motion and generate a dense warp field for stabilizing input frames. Our first approach seeks to constrain the frame warp field using a global linear transformation. In the second we designed specific metrics based on optical flow to exclude dynamic occlusions and motion boundaries. Our experiments show that our approach is more generalizable and produces both visually and quantatively better results compared to previous video stabilization methods.

We introduce RealmDreamer, a technique for generation of forward-facing 3D scenes from text descriptions. Our method optimizes a 3D Gaussian Splatting representation to match complex text prompts using pretrained diffusion models. Our key insight is to leverage 2D inpainting diffusion models conditioned on an initial scene estimate to provide low variance and high-fidelity estimates of unknown regions during 3D distillation. In conjunction, we imbue correct geometry with geometric distillation from a depth diffusion model, conditioned on samples from the inpainting model. We find that the initialization of the optimization is crucial, and provide a principled methodology for doing so. Notably, our technique does not require video or multi-view data and can synthesize a variety of high-quality 3D scenes in different styles with complex layouts. Further, the generality of our method allows 3D synthesis from a single image. As measured by a comprehensive user study, our method outperforms all existing approaches, preferred by 88-95%.

Reconstructing scenes or objects from observed images has long been a critical problem in the graphics and vision community.Traditional methods solve the inverse problem by having a large number of images to resolve the ambiguity in geometry and appearance. However, capturing and storing complex data consumes extensive compute resources, and it is infeasible for consumer-grade hardware. This dissertation presents several algorithms to reconstruct the 3D geometry and appearance from a handful of input views, allowing efficient data capture, storage and generalization to unseen scenes.

Starting with scene reconstruction, we first aim at data captured by 360° cameras.We introduce multi depth panoramas, a compact representation to enable translational and rotational movements in the 3D scene. We leverage multi-view stereo (MVS) techniques and deep neural networks to promote 16 input views into a panoramic representation that could efficiently render convincing visual results with a small storage requirement.

Furthermore, we explore a harder problem by reducing the input view count to only 2 and capturing scenes with dynamic components.We present deep 3D mask volume, a novel representation to ensure temporally stable renderings for view extrapolation. Our network takes information from the video frames to infer disocclusion caused by the moving objects. Then it produces a 3D mask volume to clean up the disoccluded regions with the temporally stable background content, producing flicker-free visual results.

Next, we focus on human portraits and seek to change the viewpoint and the lighting at the same time.We develop neural light-transport field (NeLF). This representation is trained on synthetic human portraits to generate novel views under novel lighting from only 5 input images.

Finally, we investigate the 3D reconstruction problem where only a single image is given.To this end, we present VisionNeRF. This algorithm combines the expressiveness and capacity from vision transformers and the high-fidelity rendering from volumetric representation to synthesize unseen views of a given object.