Robust human pose estimation is of particular interest to the computer vision community, and can be applied to a broad range of applications such as automated surveillance, human-computer interaction, and human activity recognition. In this dissertation, we present a framework for human pose estimation based on stochastic image grammars. Humans in particular are difficult to model, as their articulated geometry, camera viewpoint, and perspective, can produce a very large number of distinct shapes in images. Furthermore, humans often exhibit highly variant and amorphous part appearances, have self-occlusion, and commonly appear in cluttered environments. Our approach capitalizes on the reconfigurable and modular nature of grammatical models to cope with this variability in both geometry and appearance. We present a human body model as a stochastic context-sensitive AND-OR graph grammar, which represents the body as a hierarchical composition of primitive parts while maintaining the articulated kinematics between parts. Each body instance can be composed from a different set of parts and relations in order to explain the unique shape or appearance of that instance. We present grammar models based on coarse-to-fine phrase-structured grammars as well as dependency grammars, and describe efficient algorithms for learning and inference from both generative and discriminative perspectives. Furthermore, we propose extensions to our model to provide ambiguity reasoning in crowded scenes through the use of composite cluster sampling, and reasoning for self-occlusion and external occlusion of parts. We also present a technique to incorporate image segmentation into the part appearance models to improve localization performance on difficult to detect parts. Finally, we demonstrate the effectiveness of our approach by showing state-of-art performance on several recent public benchmark datasets.

The dissertation proposes a new hierarchical computational framework for social interaction understanding, addressing four core challenges - interactiveness, shared attention, gaze communication and triadic belief dynamics. The goal of the framework is to represent and model the underlying structure of social interaction, from the outside to the inside, and from the bottom to the top, and to build a bridge between typical pattern recognition in computer vision and Theory of Mind understanding in Artificial Intelligence. To achieve the goal, we firstly build a computational model for interactiveness perception from motion trajectories. Secondly, we studied the typical phenomenon shared attention, and proposed a model to temporally and spatially detect whether there is shared attention or not in the social interaction scenario, and if yes, where is the shared attention. Thirdly, in a follow up work, we extend the work to further study the general gaze communication systematically. Based on previous studies in psychology, computer vision and robotics, we summarize and define the gaze communication in two hierarchical levels, i.e., the atomic level, and the event level. We proposed a graph neural network to model what kind of gaze communication is going on in the social interaction. Finally, we step further into the underlying mental world from the appearance observations. We learn what’s the on-going belief dynamics in a triadic social interaction caused by their nonverbal communication. We jointly parse the social interaction scenario into a six-level parse graph. At the bottom are the detected entities, such as objects and human agents; the entities belong to a certain frame, and the frames are then temporally clustered into interactive segments; we recognize what is the nonverbal communication category for each interactive segment; finally, we infer the belief dynamics caused by the nonverbal communication. Our hierarchical computational framework can give a more holistic and deeper understanding of social interaction in videos beyond simple pattern perception and recognition.

For machines working alongside with humans, it is necessary to understand humans' mental states, including desires, beliefs and intentions for better interactions. In addition, humans also need to understand machines' capabilities and limitations to trust and rely on machines approriately. Achieving such bidrectional mental state alignment is crucial to the success of human-machine collaboration. This dissertation addresses this core challenge from both directions, spanning the domains of embodied artificial intelligence, autonomous driving and human-robot interaction. In the first direction of machines understanding humans, I propose a virtual environment for embodied agents and human users to work on daily activities via simulation. To enable embodied agents to better understand and execute human commands, I propose a benchmark allowing them to actively ask questions to resolve language ambiguities. For autonomous driving systems to have an accurate mental model of drivers, I propose a novel protocol to evaluate the effects of human-machine interfaces on drivers' situational awareness in different traffic conditions. In the second direction, I study how robots can generate communicative actions to be better understood by humans. I propose i) an action parsing algorithm based on an And-Or graph representation to generate explanations of task plans, ii) a task and motion planning framework to calibrate robot reachable workspace by expressive motions. My works culminate in building a computational framework for bidirectional value alignment, which is evaluated in the human-machine collaborative scout exploration game.

Theory of mind (ToM) refers to the ability to understand oneself’s and others’ mental states. The major difficulty of incorporating ToM into multi-agent planning is updating an agent’s beliefs of others’ beliefs over time, which are probability distributions over distributions. In this work, we propose a novel way to model this nested belief update with a higher computational efficiency, simultaneously providing an approach to learn other agents’ models. We model the belief update by a Markov probability transition, which linearly updates beliefs as distributions. This transition kernel characterizes another agent’s belief update and it is learnable, thus we learn other agents’ models by learning their kernels. We demonstrate the effectiveness of our algorithm in a police-thief game, showing that our agent is able to learn other agents’ belief updates and intentionally change their beliefs to achieve its own goal.

In many reinforcement learning scenarios such as many game environments or real lifesituations, the rewards are usually very limited and sparse. This kind of tasks are always difficult for agent to learn and explore. In fact, dealing with sparse reward environments has always been a challenge in reinforcement learning. In this work, we aim to study the agent driven by curiosity that can handle tasks with sparse rewards using a self-supervised method. We also want to test the possibility about agents driven by their intrinsic curiosity being able to explore the environment and generate reward by themselves. As a result, curiosity makes agents more intelligent. In the later experiments, we test two curiosity based RL methods in three different games and demonstrated that curiosity could indeed achieve very impressive performance in sparse reward game environment.

One of the main challenges of advancing task-oriented learning such as visual task planning andreinforcement learning is the lack of realistic and standardized environments for training and test-ing AI agents. Previously, researchers often relied on ad-hoc lab environments. There have beenrecent advances in virtual systems built with 3D physics engines and photo-realistic rendering forindoor and outdoor environments, but the embodied agents in those systems can only conduct sim-ple interactions with the world (e.g., walking around, moving objects, etc.). Most of the existingsystems also do not allow human participation in their simulated environments. In this work, wedesign and implement a virtual reality (VR) system, VRKitchen, with integrated functions whichi) allow human teachers to perform demonstrations to train agents (i.e., learning from demonstra-tion). ii) allow users to collect multi-modal sensor data to perform other computer vision tasks. Weprovide standardized evaluation benchmarks and data collection tools to facilitate a broad use inresearch on general computer vision research. Especially, we evaluate our collected data on humanattention prediction task.

Understanding of a incoming video sequence is one of the key parts of modern computer vision. Visualization of all the details that a computer understands about a certain video sequence is a very important tool to evaluate a vision system. In this paper, we propose a method for visualizing a 2D video sequence in a 3D space, which is based on the data of DARPA project.

The visualization started with the reconstruction of 3D space depended on the 2D video sequence data information. The use of homographic transformation for a structured light system reduced the computational cost of the system, since it only needed one image of the underlying environment. The smoothing of 3D trajectories of moving person was calculated from detection results. All the human skeleton models were from a pre-created 3D skeleton library. Aligning all the skeletons was a challenging problem in this project. We used the human-object interaction relationship and temporal information of the trajectories, such as motion information, trying to put the skeletons in the correct place. The results showed that visualization was a good way to express the understanding results of a computer vision system.

Natural scenes contain rich stochastic motion patterns which are characterized by the movement of a large number of small elements, such as falling snow, raining, flying birds, firework and waterfall. In this paper, we call these motion patterns textured motion and present a generative method that combines statistical models and algorithms from both texture and motion analysis. The generative method includes the following aspects. !). Photometrically, an image is represented as a superposition of linear bases in atomic decomposition using an over-comlete dictionary, such as Gabor or Laplacian. Such base representation is known to be generic for natural images, and it is low dimensional as the number of bases is often 100 times smaller than the number of pixels. 2). Geometricall, each moving element (called moveton), such as the individual snowflake and bird, is represented by a deformable template which is a group of several spatially adjacent bases. Such templates are learned through clustering. 3). Dynamically, the movetons are tracked through the image sequence by a stochastic algorithm maximizing a posterior probability. A classic second order Markov chain model is adopted for the motion of dynamics. The sources and sinks of the movetons are modeled by birth and death maps. We adopt an EM-like stochastic gradient algorithm for inference of the hidden variables: bases, movetons, birth/death maps, parameters of the dynamics. The learned models are also verified through synthesizing random textured motion sequences which bear similar visual appearances with the ovserved sequences.

We propose a robust multi-modal method for automatically registering a moving camera (e.g. mounted on a robot) in 3D scene map. Our approach takes advantages of both Global Positioning System (GPS) and visual sensors to obtain high-precision geographic location for a moving camera at each time. The proposed method distinguishes from past works in the following three aspects: i) we introduce a spatial pyramid mesh warping method to obtain dense correspondences between consecutive frames, which can be used to remove unexpected camera motion for robust registration ; ii) we introduced a robust feature tracking method to tracking feature points in consecutive frames; and iii) we utilize a continuous polynomial function to describe camera motion w.r.t time, which can be solved by minimizing the errors of interpolating both visual observations and GPS locations. We evaluate the proposed method on a set of challenging videos for both stabilization and registration tasks. Results with comparisons to other popular methods showed that our method is capable of achieving high-quality results under various challenges, e.g. lighting changes, motion blurs, scene noises etc.

Humans can extract rich information from visual scenes, such as the 3D locations of objects and humans, the actions of humans, the states of objects, the belief of humans. Although various state-of-the-art algorithms can achieve good results for solving individual tasks, building a system to jointly infer these different tasks for scene understanding is still an underexplored area. Most of these tasks are not independent with each other, and humans can jointly infer hidden information with their commonsense knowledge among these tasks. In this dissertation, we propose a spatio-temporal framework to jointly infer and optimize multiple tasks across different times and views with a unified explicit probabilistic graphical representation.

This dissertation contains four main parts. 1) we describe the system overview, the data flow in the system, and engineering efforts to make the system scalable under different scenarios. 2) we propose an algorithm for holistic 3D scene parsing and human pose estimation with human-object interaction and physical commonsense. Human-object interaction can model the fine-grained relations between agents and objects, and physical commonsense can model the physical plausibility of the reconstructed scene. 3) we introduce a joint parsing framework that integrates view-centric proposals into scene-centric parse graphs that represent a coherent scene-centric understanding of cross-view scenes. 4) we present a joint inference algorithm to understanding object states, robot knowledge, and human beliefs under multi-view settings by maintaining three types of parse graphs. The algorithm can be applied to the cross-view small object tracking problem and some false-belief problems. Experiments show that our joint inference framework can achieve better results than individual algorithms.