UC Berkeley

Multi-agent interacting systems are prevalent in the world, from purely physical systems to complicated social dynamic systems. The interactions between entities or components can give rise to very complex behavior patterns at the level of both individual and the whole multi-agent system. The involved entities in these systems need to interact with each other and their behaviors tend to have mutual influence. They need to perceive and behave compliant to the physical environment as well. This brings the necessity of effective reasoning on the agent-agent and agent-context relations/interactions. This especially plays a significant role in safety-critical applications such as autonomous driving and social robot navigation.

Technically, it is challenging to model the dynamics of multi-agent interacting systems due to the internal heterogeneity of agents, uncertainty and multi-modality in the future behavior, evolution or change of the context, etc. The fundamental research question to address in this dissertation is how to model the multi-agent relations and interactions in a unified, generalizable framework with an effective relational representation.

The focus of this dissertation is 1) to design a multi-agent behavior modeling framework with relational reasoning in dynamically evolving uncertain environments for heterogeneous agents; 2) to design a generic importance estimation framework with relational reasoning for scene understanding.

This dissertation is divided into two parts. Part I focuses on the multi-agent prediction and tracking problems. In Chapter 2, a hierarchical time-series prediction model is introduced for situation and behavior recognition based on probabilistic graphical models, which can be applied to the scenarios with a single autonomous agent or under a fixed multi-agent setting. In Chapter 3, deep generative modeling techniques are employed to learn the data distribution, which can generate more diverse and realistic prediction hypotheses. In Chapter 4, a graph representation is further leveraged to capture spatio-temporal interaction patterns, which is a natural way to represent multiple agents in the scene and their relations. Different from the method discussed in Chapter 4 where the graph topology is determined by distance-based heuristics, in Chapter 5 we propose to learn a latent relational graph structure from observation data, which can evolve over time to enable dynamic relational reasoning. An accurate prediction model plays a significant role in multi-target tracking frameworks, especially in highly dynamic and interactive scenarios. In Chapter 6, a unified tracking and prediction framework based on a modified sequential Monte Carlo method is discussed, which can adopt any of the above prediction models as the implicit proposal distribution. Part II addresses another related downstream problem (i.e., importance estimation) of relational reasoning under a multi-agent setting. In Chapter 7, a hybrid attention inference network is presented to recognize relative importance of objects in the scene based on observation data, which enables dynamic key information selection. In Chapter 8, we further investigate how to inject human knowledge by proving human annotations with a self-supervised learning pipeline, which enables the model to learn from unlimited, unlabeled data.