Real world scenes and objects have diverse visual appearance. Such diversity stems from the fundamental physics in how light interacts with matter, across different weather conditions, object types, and even people. These appearance variations mesmerize human beings, but puzzle artificial vision systems, which cannot generalize to such diversity. Through this thesis, we look at one such case of biased performance over diversity- camera based remote heart rate (HR) estimation. HR is an essential clinical measure for the assessment of cardiorespiratory instability. The growing telemedicine market opens up the urgent requirement for scalable yet affordable remote HR estimation. However, existing computer vision methods that estimate HR from facial videos exhibit biased performance against dark skin tones. This is a major concern, since communities of color are disproportionately affected by both COVID-19 and cardiovascular disease. We identify and model the origin of this bias and present a novel physics-driven algorithm that boosts performance on darker skin tones in our reported data. We assess the performance of our method through the creation of the first telemedicine-focused remote vital signs dataset, the VITAL dataset. 432 videos (~864 minutes) of 54 subjects with diverse skin tones are recorded under realistic scene conditions with corresponding vital sign data. Our method mitigates errors due environmental conditions and imparts unbiased performance gains across skin tones, setting the stage for making non-contact HR sensing technologies a viable reality for patients across skin tones.

Shape from Polarization (SfP) recovers an object's shape from polarized photographs of the scene. In previous works, the SfP algorithms use idealized physical equations to recover the shape. These previous approaches are error-prone when real-world conditions deviate from the idealized physics. In this thesis, we propose a physics-based neural network to address the SfP problem. Our algorithm fuses deep learning with synthetic renderings (derived from physics) to exceed the quality of all previous SfP methods. A two-stage encoder is used to resolve the longstanding problem of ambiguities. Our results of surface normal recovery are an improvement upon methods that utilize physics-based solutions alone.

Non-line-of-sight (NLOS) imaging has relevance in search

Heart rate (HR) is an essential clinical measure for the assessment of cardiorespiratory instability. The growing telemedicine market opens up the urgent requirement for scalable yet affordable remote HR estimation. Smartphones that use in-built camera modules to measure HR from facial videos offer a more economical solution in comparison to mass deployment of wearable sensors. However, existing computer vision methods that estimate HR from facial videos exhibit biased performance against dark skin tones. This is a major concern, since communities of color are disproportionately affected by both COVID-19 and cardiovascular disease. We identify the origin of this bias and present a novel physics-driven algorithm that boosts performance on darker skin tones in our reported data. We assess the performance of our method through the creation of the first telemedicine-focused remote vital signs dataset, the VITAL dataset. 472 videos (~944 minutes) of 59 subjects with diverse skin tones are recorded under realistic scene conditions with corresponding vital sign data. Our method reduces errors due to lighting changes, shadows, and specular highlights and imparts unbiased performance gains across skin tones, setting the stage for making non-contact HR sensing technologies a viable reality for patients across skin tones, using just smartphone cameras.

This thesis attempts at discovering features from visual data which cannot be obtained or observed directly using standard computer vision algorithms. We refer to these features as Invisible features. In particular, we focus on two types of invisible features. First, the physics of a scene which governs the visual cues for the objects in the scene. In this part of the thesis, we teach a machine to discover the laws of physics from video streams. We assume no prior knowledge of physics, beyond a temporal stream of bounding boxes. The problem is very difficult because a machine must learn not only a governing equation (e.g. projectile motion) but also the existence of governing parameters (e.g. velocities). Second, texture information that is invisible in standard RGB images but can be seen in other imaging modalities such as polarization. Texture in some scenes is challenging for intensity images to capture. Imagine a black car in shadow or an oil slick on road. We exploit this adversity of contrast in the intensity domain to adapt a new representation for polarization cues, proposing a new degree of linear polarization (DOLP) that has favorable statistical properties. The new representation of DOLP we obtain is not only more robust in the context of noise, but can also preserve the scientific information in the original DOLP that allows geometry and photoelastic effects to be discerned. We hope this work lays a foundation for the future of a Polarized ISP process, particularly for sensor fusion applications.

Deep learning has exhibited remarkable performance on various computer vision tasks. However, these models usually suffer from the generalization issue when the training sets are not sufficiently large or diverse. Human intelligence, on the other hand, is capable of learning with a few samples. One of the potential reasons for this is that we use other prior knowledge to generalize to new environments and unseen data, as opposed to learning everything from the provided training sets. We aim to enable machines with such capability. More specifically, we focus on integrating different types of prior physical knowledge and inductive biases into neural networks for various computer vision applications.

The core idea is to exploit physical models as inductive biases and design specific strategies to blend them with the neural network learning process. This problem is difficult since we need to consider both the fidelity of our prior knowledge and the quality of the training samples. To validate the effectiveness of the proposed blending strategies, extensive experiments have been conducted on multiple computer vision tasks, such as Shape from Polarization (SfP), remote photoplethysmography (rPPG), and single-image rain removal.

Camera-based remote photoplethysmography (rPPG) provides a non-contact way to measure physiological signals (e.g., heart rate) using facial videos. Recent deep learning architectures have improved the accuracy of such physiological measurement significantly, yet they are restricted by the diversity of the annotated videos. The existing datasets MMSE-HR, AFRL, and UBFC-RPPG contain roughly 10%, 0%, and 5% of dark-skinned subjects respectively. The unbalanced training sets result in a poor generalization capability to unseen subjects and lead to unwanted bias toward different demographic groups. Here we show a first attempt to overcome the lack of dark-skinned subjects by synthetic augmentation. A joint optimization framework is utilized to translate real videos from light-skinned subjects to dark skin tones while retaining their pulsatile signals. In the experiment, our method exhibits around 31% reduction in mean absolute error for the dark-skinned group and 46% improvement on bias mitigation for all the groups, as compared with the previous work trained with just real samples.

The COVID-19 pandemic brought telemedicine applications under the spotlight and the heart rate (HR) is an important clinical vital sign in the evaluation of cardiorespiratory and hemodynamic stability. However, both deep learning- and signal processing-based systems demonstrate biased performance towards dark skin tones in remote HR measurements, as performance measures are restricted to the diversity of dataset subjects. The existing datasets MMSE-HR, AFRL, and UBFC-RPPG contain roughly 10%, 0%, and 5% dark-skinned subjects respectively, leading to poor generalization capability to unseen subjects and lead to unwanted bias toward different demographic groups. We propose a physics-driven algorithm to combat this bias and show a first attempt to overcome the lack of dark-skinned subjects by synthetic augmentation. A joint optimization framework is utilized to translate real videos from light-skinned subjects to dark skin tones while retaining their pulsatile signals. In the experiment, our method exhibits around 31% reduction in mean absolute error for the dark-skinned group and 46% improvement on biasm itigation for all the groups, as compared with previous work trained with just real samples.

Teaching machines to speak and act like a human is one of the longest-running goals in Artificial Intelligence. This thesis tackles two important problems in building the next generation of dialogue systems: enhancing the emotional intelligence of social chatbots and learning semantic structures from dialogue corpus. On the one hand, taking into account emotional quotient in dialogue system design help machine to mimic human behavior and further improve the long-term user engagement. On the other hand, extracting structural information from dialogue data is critical for us to analyze user behavior and system performance. The technology could be applied to various areas in computational linguistics, such as dialogue management, discourse analysis, and dialogue summarization.

This thesis consists of two parts. In the first part, we aim to present our efforts at studying emotional intelligence in dialogues systems. We break down the problem into three subjects: 1) the modeling and incorporation of human values, i.c., people tend to have common attitudes towards some statements or scenarios; 2) the inference of social relations between interlocutors from dialogues. Chatbots with such inferring capability can understand human behavior better and act appropriately; and 3) the modeling and tracking of speakers' mental states. This is beyond understanding what users say to perceive what users imply, requiring agents to mentally simulate the evolution status of the environment.

In the second part of this thesis, we investigate how we can extract structural information from dialogue corpus. In particular, we pioneered two research directions: 1) we reconstruct the original dialogues with variational recurrent neural networks and structured attention, then we extract the structure by computing the transitions between latent states; and 2) we detect and track the status of potential slot token groups to approximate a representation of task-oriented dialogue structures. We explored the problem from both theoretical and practical perspectives.