Humans can extract rich information from visual scenes, such as the 3D locations of objects and humans, the actions of humans, the states of objects, the belief of humans. Although various state-of-the-art algorithms can achieve good results for solving individual tasks, building a system to jointly infer these different tasks for scene understanding is still an underexplored area. Most of these tasks are not independent with each other, and humans can jointly infer hidden information with their commonsense knowledge among these tasks. In this dissertation, we propose a spatio-temporal framework to jointly infer and optimize multiple tasks across different times and views with a unified explicit probabilistic graphical representation.

This dissertation contains four main parts. 1) we describe the system overview, the data flow in the system, and engineering efforts to make the system scalable under different scenarios. 2) we propose an algorithm for holistic 3D scene parsing and human pose estimation with human-object interaction and physical commonsense. Human-object interaction can model the fine-grained relations between agents and objects, and physical commonsense can model the physical plausibility of the reconstructed scene. 3) we introduce a joint parsing framework that integrates view-centric proposals into scene-centric parse graphs that represent a coherent scene-centric understanding of cross-view scenes. 4) we present a joint inference algorithm to understanding object states, robot knowledge, and human beliefs under multi-view settings by maintaining three types of parse graphs. The algorithm can be applied to the cross-view small object tracking problem and some false-belief problems. Experiments show that our joint inference framework can achieve better results than individual algorithms.

Robust pose estimation and attribute classification are of particular interest and important tasks to the computer vision community, and they are frequently used as the intermediate representations for other high-level tasks, such as human identification, visual search and human tracking. In this dissertation, we present a unified framework for joint inferring human body pose and human attribute in a parse graph with attributes augmented to nodes in the hierarchical representation. Particularly, unlike previous existing approaches mostly train models for the two tasks separately and combine the inference sequentially, we build a unified framework by integrating the three traditional grammar formulations in an And-Or graph representation: (i) Phrase structure grammar representing the hierarchical decomposition of the human body; (ii) Dependency grammar modeling the geometric articulation; and (iii) Attribute grammar accounting for the compatibility relations between different parts in the hierarchy. Furthermore, we propose extension of our model to integrate the deep learned features efficiently to provide the better performance with richer and deeper appearance representation. We also propose a technique to handle large variation of appearance and geometry. Particularly, unlike previous approaches define the parts by drawing square bounding box around keypoints or annotating precise bounding box for parts, our approach defines parts through the separate part proposal process. Finally, we demonstrate the effectiveness of both of joint modeling and integrating deep models by showing state-of-the-art performance on several recent public benchmark datasets. We also collect our own dataset and compare our approach with existing methods.

It is a challenging yet crucial task to have a comprehensive understanding of human activities and events in the 3D scene. This task involves many many mid-level vision tasks (e.g., detection, tracking, pose estimation, action/interaction recognition) and requires high-level understandings and reasoning about their relations. In this dissertation, we aim to propose a novel and general framework for both mid-level and high-level tasks under this track, towards a better solution for complex 3D scene and event understanding. Specifically, we aim to formulate problems with interpretable representations, enforce high-level constraints with domain knowledge guided grammar, learn models solving multiple tasks jointly, and infer based on spatial, temporal and casual information. We make three major contributions in this dissertation:

First, we introduce interpretable representations to incorporate high-level constraints defined by domain knowledge guided grammar. Specifically, we propose: i) Spatial and Temporal Attributed Parse Graph model (ST-APG) encoding compositionality and attribution for multi-view people tracking, enhancing trajectory associations across space and time, ii) Scene-centric Parse Graph to represent a coherent understanding of information obtained from cross-view scenes for multi-view knowledge fusion, iii) Fashion Grammar for constraining configurations of human appearance and clothing in human parsing, iv) Pose Grammar for describing physical and physiological relations among human body parts in human pose estimation, and v) Causal And-Or Graph (C-AOG) to represent the causal-effect relations between an object's fluent changes and involved activities in tracking interacting objects.

Second, we formulate multiple related tasks into a joint learning, inference and reasoning framework for mutual benefits and better configurations, instead of solving each task independently. Specially, we propose: i) a joint parsing framework for iteratively tracking people locations and estimating people attributes, ii) a joint inference framework modeled by deep neural networks for passing messages from direct, top-down and bottom-up directions in the task of human parsing, and iii) a joint reasoning framework to reason object's fluent changes and track the object in videos, iteratively searching for a feasible causal graph structure.

Third, we mitigate the problem of data scarcity and data-hungry model learning using a learning-by-synthesis framework. Given limited training samples, we consider either propagate supervisions to unpaired samples or synthesizing virtual samples that minimize discrepancies with the realistic data. Specifically, we develop a pose sample simulator to augment training samples in virtual camera views for the task of 3D pose estimation, which improves our model cross-view generalization ability.

There are several interesting properties regarding the proposed frameworks: i) a novel perspective for problem formulation on joint inference and reasoning on space, time and causality, ii) overcoming the drawbacks of lack of interpretability and data hunger for end-to-end deep learning methods. Experiments show that our joint inference and reasoning framework outperforms existing approaches on many tasks and obtains more interpretable results.

Cogito, ergo sum. Building machines that can think and reason like humans is a long-standing goal of AI. Despite the tremendous progress in AI we witnessed in recent years, it is still not clear whether these learning machines at scale can solve problems that require sophisticated thinking and reasoning, especially when the problems are also tied to ontologies (entities, relations) in the real world and raw sensory observations, i.e., visual and relational reasoning. Further, human-level reasoning and thinking also call for the capability of generalizing what the machine has learned to problem instances with their novel forms and combinations. We anticipate such generalization should be possible even with few data as well as on diverse modalities, e.g. vision, text, embodied 3D scenes, etc, which creates a significant gap between humans and machines.

This dissertation studies human-like visual and relational reasoning in the real world, aiming at closing the aforementioned gap between humans and machines. The first part of this thesis focuses on deepening the current understanding of the limitations of existing ML-based reasoning systems when compared to humans. To this end, a series of benchmarks are developed in hope of examining the full spectrum of anticipated capabilities of these systems, including zero-shot, few-shot generalization, and adaptation to difficult modalities including embodied 3D scenes. Based off these new quests for AI reasoning, thorough evaluations are conducted with recently proposed reasoning systems, and their limitations are discussed.

The second part of this dissertation introduces a unified framework by drawing inspiration from the human language system, which is grounded, entity-centric, semantically rich, and could be the key to human-level generalization in reasoning. Specifically, the problem of learning language-like representations from a generative learning perspective is investigated. The resulting models can facilitate learning object-centric representations from images and discrete-continuous hybrid representations from text using an energy-based formulation. Further, intuitive and scalable inductive biases are developed to leverage the semantic supervision from the English language to learn object-centric and relational representations, to directly tackle the challenging zero-shot systematic generalization problem in visual and relational reasoning. Finally, what could be the next major move in the field is highlighted.

Theory of mind (ToM) refers to the ability to understand oneself’s and others’ mental states. The major difficulty of incorporating ToM into multi-agent planning is updating an agent’s beliefs of others’ beliefs over time, which are probability distributions over distributions. In this work, we propose a novel way to model this nested belief update with a higher computational efficiency, simultaneously providing an approach to learn other agents’ models. We model the belief update by a Markov probability transition, which linearly updates beliefs as distributions. This transition kernel characterizes another agent’s belief update and it is learnable, thus we learn other agents’ models by learning their kernels. We demonstrate the effectiveness of our algorithm in a police-thief game, showing that our agent is able to learn other agents’ belief updates and intentionally change their beliefs to achieve its own goal.

The study of human-object interaction (HOI) aims at modeling the geometric relationship between a human and an object in an interaction. Understanding HOI is an essential step towards holistic scene understanding and generating realistic scenarios that involve humans. Conventionally, the study of HOI focuses on detecting and classifying instance-level HOI on 2D images. Given an image, an example output would be a triplet , or , where the person, chair, and apple are all represented by bounding boxes. This dissertation aims to understand HOI in 3D.

Extending HOI to 3D faces two significant challenges. The first challenge lies in the difficulty of obtaining high-fidelity 3D annotation of HOI data. Existing methods of collecting 3D datasets all suffer from high occlusion, poor resolution, and high annotation costs. Another critical challenge in modeling 3D HOI lies in the representation of the objects. Existing methods treat each object as a unity, usually represented as an axis-aligned bounding box. Such methods ignore the complexity of objects' shapes and therefore fail to model complex geometrical relationships in HOIs such as sitting. The root cause of this challenge traces back to the first challenge, where we do not have the high-fidelity data necessary to reflect the details in object shapes.

This dissertation addresses both challenges by collecting a large-scale high-fidelity 3D HOI dataset and by proposing hierarchical modeling of HOI. By using instance-level HOI annotation, our dataset improves scene reconstruction performance by a significant margin. This high-fidelity nature of the collected dataset enables part-level HOI modeling, which addresses the second challenge. This dissertation also addresses the second challenge by decomposing shape-level HOI into physics-level, which significantly improves the quality and robustness of grasp synthesis.

Scene understanding and event understanding of humans correspond to the spatial and temporal aspects of computer vision. Such abilities serve as a foundation for humans to learn and perform tasks in the world we live in, thus motivating a task-oriented representation for machines to interpret observations of this world.

Toward the goal of task-oriented scene understanding, I begin this thesis by presenting a human-centric scene synthesis algorithm. Realistic synthesis of indoor scenes is more complicated than neatly aligning objects; the scene needs to be functionally plausible, which requires the machine to understand the tasks that could be performed in the scene.

Instead of directly modeling the object-object relationships, the algorithm learns the human-object relations and generate scene configurations by imagining the hidden human factors in the scene. I analyze the realisticity of the synthesized scenes, as well as its usefulness for various computer vision tasks. This framework is useful for backward inference of 3D scenes structures from images in an analysis-by-synthesis fashion; it is also useful for generating data to train various algorithms.

Moving forward, I introduce a task-oriented event understanding framework for event parsing, event prediction, and task planning. In the computer vision literature, event understanding usually refers to action recognition from videos, i.e., "what is the action of the person". Task-oriented event understanding goes beyond this definition to find out the underlying driving forces of other agents. It answers questions such as intention recognition ("what is the person trying to achieve"), and intention prediction ("how the person is going to achieve the goal"), from a planning perspective.

The core of this framework lies in the temporal representation for tasks that is appropriate for humans, robots, and the transfer between these two. In particular, inspired by natural language modeling, I represent the tasks by stochastic context-free grammars, which are natural choices to capture the semantics of tasks, but traditional grammar parsers (e.g., Earley parser) only take symbolic sentences as inputs. To overcome this drawback, I generalize the Earley parser to parse sequence data which is neither segmented nor labeled. This generalized Earley parser integrates a grammar parser with a classifier to find the optimal segmentation and labels. It can be used for event parsing, future predictions, as well as incorporating top-down task planning with bottom-up sensor inputs.

We propose a robust multi-modal method for automatically registering a moving camera (e.g. mounted on a robot) in 3D scene map. Our approach takes advantages of both Global Positioning System (GPS) and visual sensors to obtain high-precision geographic location for a moving camera at each time. The proposed method distinguishes from past works in the following three aspects: i) we introduce a spatial pyramid mesh warping method to obtain dense correspondences between consecutive frames, which can be used to remove unexpected camera motion for robust registration ; ii) we introduced a robust feature tracking method to tracking feature points in consecutive frames; and iii) we utilize a continuous polynomial function to describe camera motion w.r.t time, which can be solved by minimizing the errors of interpolating both visual observations and GPS locations. We evaluate the proposed method on a set of challenging videos for both stabilization and registration tasks. Results with comparisons to other popular methods showed that our method is capable of achieving high-quality results under various challenges, e.g. lighting changes, motion blurs, scene noises etc.

To allow wide-spread adoption of consumer robotics, robots must be able to adapt to their

environment by learning new skills and communicating with humans. Each chapter explains a

contribution to achieve this goal. Chapter One covers a stochastic And-Or knowledge

representation framework for robotic manipulations. Chapter Two further expands this

established system for robustly learning from perception. Chapter Three unifies perception with

natural language for a joint real-time processing of information. We've successfully tested the

generalizability and faithfulness of our robotic knowledge acquisition and inference pipeline. We

present proof of concepts in each of the three chapters.

The dissertation proposes a new hierarchical computational framework for social interaction understanding, addressing four core challenges - interactiveness, shared attention, gaze communication and triadic belief dynamics. The goal of the framework is to represent and model the underlying structure of social interaction, from the outside to the inside, and from the bottom to the top, and to build a bridge between typical pattern recognition in computer vision and Theory of Mind understanding in Artificial Intelligence. To achieve the goal, we firstly build a computational model for interactiveness perception from motion trajectories. Secondly, we studied the typical phenomenon shared attention, and proposed a model to temporally and spatially detect whether there is shared attention or not in the social interaction scenario, and if yes, where is the shared attention. Thirdly, in a follow up work, we extend the work to further study the general gaze communication systematically. Based on previous studies in psychology, computer vision and robotics, we summarize and define the gaze communication in two hierarchical levels, i.e., the atomic level, and the event level. We proposed a graph neural network to model what kind of gaze communication is going on in the social interaction. Finally, we step further into the underlying mental world from the appearance observations. We learn what’s the on-going belief dynamics in a triadic social interaction caused by their nonverbal communication. We jointly parse the social interaction scenario into a six-level parse graph. At the bottom are the detected entities, such as objects and human agents; the entities belong to a certain frame, and the frames are then temporally clustered into interactive segments; we recognize what is the nonverbal communication category for each interactive segment; finally, we infer the belief dynamics caused by the nonverbal communication. Our hierarchical computational framework can give a more holistic and deeper understanding of social interaction in videos beyond simple pattern perception and recognition.