The problem of novel view synthesis has grown significantly in popularity recently with the introduction of Neural Radiance Fields (NeRFs) and other implicit scene representation methods. A recent advance, 3D Gaussian Splatting (3DGS), leverages an explicit representation to achieve real-time rendering with high-quality results. However, 3DGS still requires an abundance of training views to generate a coherent scene representation. In few shot settings, similar to NeRF, 3DGS tends to overfit to training views, causing background collapse and excessive floaters, especially as the number of training views are reduced. This work proposes a method to enable training coherent 3DGS-based radiance fields of 360° scenes from sparse training views. Depth priors are integrated with generative and explicit constraints to reduce background collapse, remove floaters, and enhance consistency from unseen viewpoints. Experiments show that this method outperforms base 3DGS by 6.4% in LPIPS and by 12.2% in PSNR, and NeRF-based methods by at least 17.6% in LPIPS on the MipNeRF-360 dataset with substantially less training and inference cost. Project website at: https://tinyurl.com/sparsegs.

Contactless vital sensing is gaining prominence with applications in disease control, health monitoring, and medicine; airports are beginning to use infrared thermometers to screen for fevers, automobile companies are researching how cars can wirelessly detect drowsy drivers, and the medical field is exploring the benefits of how cameras can be used to remotely monitor neonates or detect diseases such as atrial fibrillation or sleep apnea. However, prior research to a large extent has not explored when remote vital sensing methods fail and if they may be disadvantageous to certain physiologies more than others such as age, weight, or gender. New methods in the field should strive to determine the impact of these variables as well as rectify inaccuracies in sensing that may occur if possible. This work explores how skin tone can adversely impact heart-rate detection with cameras and temperature evaluation with thermal cameras. Multimodal fusion and algorithmic techniques are proposed to improve skin tone equity while improving performance of contactless vital sensing methods.

Deep learning has exhibited remarkable performance on various computer vision tasks. However, these models usually suffer from the generalization issue when the training sets are not sufficiently large or diverse. Human intelligence, on the other hand, is capable of learning with a few samples. One of the potential reasons for this is that we use other prior knowledge to generalize to new environments and unseen data, as opposed to learning everything from the provided training sets. We aim to enable machines with such capability. More specifically, we focus on integrating different types of prior physical knowledge and inductive biases into neural networks for various computer vision applications.

The core idea is to exploit physical models as inductive biases and design specific strategies to blend them with the neural network learning process. This problem is difficult since we need to consider both the fidelity of our prior knowledge and the quality of the training samples. To validate the effectiveness of the proposed blending strategies, extensive experiments have been conducted on multiple computer vision tasks, such as Shape from Polarization (SfP), remote photoplethysmography (rPPG), and single-image rain removal.

This dissertation advances subtle realism in digital representations by developing methods forphysiologically-aware video synthesis, efficient 3D reconstruction, and temporally consistent 4D modeling. We propose a scalable framework that generates bio-realistic face videos by preserving physiological signals, addressing demographic bias in remote health sensing. In 3D reconstruction, we introduce ALTO, a method that alternates between latent topologies to achieve high-fidelity shape recovery with fast inference. Extending to 3D generation, we present a multi-view diffusion model (MVDD) that synthesizes detailed 3D shapes from multi-view depth maps, improving upon point-based generative models. Finally, we develop a framework for dynamic 4D surface reconstruction from monocular video, ensuring temporal coherence for applications like simulation and editing. Collectively, these contributions form a cohesive progression toward realistic, scalable digital modeling of the physical world, with applications across healthcare, graphics, and AI-driven simulation. Future directions include training-free 3D reasoning, real-time dynamic modeling, and broader cross-modal generative modeling.

This thesis attempts at discovering features from visual data which cannot be obtained or observed directly using standard computer vision algorithms. We refer to these features as Invisible features. In particular, we focus on two types of invisible features. First, the physics of a scene which governs the visual cues for the objects in the scene. In this part of the thesis, we teach a machine to discover the laws of physics from video streams. We assume no prior knowledge of physics, beyond a temporal stream of bounding boxes. The problem is very difficult because a machine must learn not only a governing equation (e.g. projectile motion) but also the existence of governing parameters (e.g. velocities). Second, texture information that is invisible in standard RGB images but can be seen in other imaging modalities such as polarization. Texture in some scenes is challenging for intensity images to capture. Imagine a black car in shadow or an oil slick on road. We exploit this adversity of contrast in the intensity domain to adapt a new representation for polarization cues, proposing a new degree of linear polarization (DOLP) that has favorable statistical properties. The new representation of DOLP we obtain is not only more robust in the context of noise, but can also preserve the scientific information in the original DOLP that allows geometry and photoelastic effects to be discerned. We hope this work lays a foundation for the future of a Polarized ISP process, particularly for sensor fusion applications.

The problem of novel view synthesis has grown significantly in popularity recently withthe introduction of Neural Radiance Fields (NeRFs) and other implicit scene representation methods. A recent advance, 3D Gaussian Splatting (3DGS), leverages an explicit representation to achieve real-time rendering with high-quality results. However, 3DGS still requires an abundance of training views to generate a coherent scene representation. In few shot settings, similar to NeRF, 3DGS tends to overfit to training views, causing background collapse and excessive floaters, especially as the number of training views are reduced. This work proposes a method to enable training coherent 3DGS-based radiance fields of 360° scenes from sparse training views. Depth priors are integrated with generative and explicit constraints to reduce background collapse, remove floaters, and enhance consistency from unseen viewpoints. Experiments show that this method outperforms base 3DGS by 6.4% in LPIPS and by 12.2% in PSNR, and NeRF-based methods by at least 17.6% in LPIPS on the MipNeRF-360 dataset with substantially less training and inference cost. Project website at: https://tinyurl.com/sparsegs.

Real world scenes and objects have diverse visual appearance. Such diversity stems from the fundamental physics in how light interacts with matter, across different weather conditions, object types, and even people. These appearance variations mesmerize human beings, but puzzle artificial vision systems, which cannot generalize to such diversity. Through this thesis, we look at one such case of biased performance over diversity- camera based remote heart rate (HR) estimation. HR is an essential clinical measure for the assessment of cardiorespiratory instability. The growing telemedicine market opens up the urgent requirement for scalable yet affordable remote HR estimation. However, existing computer vision methods that estimate HR from facial videos exhibit biased performance against dark skin tones. This is a major concern, since communities of color are disproportionately affected by both COVID-19 and cardiovascular disease. We identify and model the origin of this bias and present a novel physics-driven algorithm that boosts performance on darker skin tones in our reported data. We assess the performance of our method through the creation of the first telemedicine-focused remote vital signs dataset, the VITAL dataset. 432 videos (~864 minutes) of 54 subjects with diverse skin tones are recorded under realistic scene conditions with corresponding vital sign data. Our method mitigates errors due environmental conditions and imparts unbiased performance gains across skin tones, setting the stage for making non-contact HR sensing technologies a viable reality for patients across skin tones.

Non-line-of-sight (NLOS) imaging has relevance in search

Generative diffusion models have recently become extremely popular in a variety of domains,but especially in image generation. These models are capable of generating a wide variety of high-quality images and can be guided by text prompts, depth maps, and more. Despite these impressive capabilities, these models typically generate images with poor projective geometry. As a result, generated images differ significantly from real images, decreasing the photo-realism of generated images. In addition, since perspective is crucial for representing 3D information in 2D images, discrepancies in projective geometry limit the use of generative models as synthetic data generators. In this work, we introduce a geometric constraint to improve the projective geometry of diffusion models and show that outputs of models trained with this constraint both appear more photo-realistic and serve as useful synthetic data by improving the performance of downstream models fine-tuned on generated images.

Visual illusions have always been a highly popular topic among the general public. People enjoy the experience of having their eyes deceived by those fascinating visual arts. Mean- while, in the computer vision community, the recent boom in deep neural networks has proven to be a powerful tool for various vision tasks, ranging from image classification and depth estimation to three-dimensional scene reconstruction. If the ultimate goal for a neural network is to perceive scenes and objects as humans do, we can’t help but wonder: can neural networks also be tricked by visual illusions? In this work, we explore how Zero-1-to-3, a 3D scene reconstruction model, can be deceived by a special category of visual illusions: shading illusions.