Background

Results

Conclusions

A major limitation on performing detailed behavioral analysis of honey bee colonies is that there is currently no efficient way to carry it out. Due to the time required in manually analyzing the data, the current approach and small sample sizes limit the statistical power of these analyses. An automated system can provide a breakthrough in the way this research is performed. Waggle dances are an important aspect of understanding the behavior of honey bees as it serves as a way to communicate among themselves. In this thesis, we develop an automated system using computer vision and learning techniques to solve two problems i) Single bee tracking and waggle detection and ii) Multiple bee waggle detection. Our approach shows that it is possible to train learning algorithms to detect when and where a waggle happens in the hive.

For successful integration of autonomous systems such as drones and self-driving cars in our day-to-day life, they must be able to quickly adapt to ever changing environments, and actively reason about their safety and that of other users and autonomous systems around them. Even though control theoretic approaches have been used for decades now for the control and safety analysis of autonomous systems, these approaches typically operate under the assumption of a known system dynamics model and the environment in which the system is operating. To overcome these challenges, machine learning approaches have been explored to operate autonomous systems intelligently and reliably in unpredictable environments based on prior data. However, learning techniques widely used today are extremely data inefficient, making it challenging to apply them to real-world physical systems. Moreover, unlike control theoretic approaches, these techniques lack the necessary mathematical framework to provide guarantees on correctness, causing safety concerns as data-driven physical systems are integrated in our society. 

This dissertation aims to combine the control theoretic perspective with the modern learning approaches to enable autonomous systems to safely adapt to unknown environments. It first introduces a suite of core tools from dynamical system theory and robust control that permits the safety analysis of single and multi-agent autonomous systems under the assumption of known system model and environment. In the remainder of the dissertation, we discuss how we can go past these assumptions with the help of machine learning, while minimizing the data requirement for learning and maintaining the safety guarantees for the autonomous system.

To that end, we first discuss how we can learn inaccuracies in the dynamics model of the system, such as unknown ground effects for quadrotors, and use the learned model along with optimal control tools to improve the control performance. Our particular focus is on learning task-specific models that allow for a quick adaptation for the task at hand; these techniques are showcased on physical quadrotors and robotic arms. We next present modular architectures that use a learning-based perception module for the environment level reasoning and a dynamics model-based module for system level reasoning to solve challenging perception and control problems in a priori unknown environments in a data-efficient fashion. Moreover, due to their modularity, these architectures are amenable to simulation-to-real transfer, and can be used for different robotic systems without any retraining. These approaches are demonstrated on a variety of ground robots navigating in unknown buildings around humans based only on onboard visual sensors.

Next, we discuss how we can use dynamics models not only for data-efficient learning, but also to monitor and recognize the learning system’s failures, and to provide online corrective safe actions when necessary. This allows us to provide safety assurances for learning-enabled systems in unknown and human-centric environments, which has remained a challenge to date. Together these techniques enable autonomous systems to learn to operate in unknown environments, but do so in a data-efficient and safe fashion. The dissertation ends with a discussion of future challenges and opportunities at the intersection of learning and control, including the safety analysis in online learning settings and the need to close the loop between the design of learning systems and their safety analysis for developing resilient and continually improving autonomous systems.

Research in natural language processing (NLP) is aimed at making machines automatically understand natural language. Incorporating world knowledge semantics is becoming an increasingly crucial requirement for resolving deep, complex decisions in most NLP tasks today, e.g., question answering, syntactic parsing, coreference resolution, and relation extraction. Structured NLP corpora such as treebanks are too small to encode much of this knowledge, so instead, we turn to the vast Web, and access its information via a diverse collection of Web n-gram counts (of size 4 billion, and ~500x larger than Wikipedia). Shallow cues from this large n-gram dataset, when harnessed in a structured learning setting, help reveal deep semantics.

In this thesis, we address various important facets of the semantics problem - from indirect semantics for sentence-level syntactic ambiguities, and semantics as specific knowledge for discourse-level coreference ambiguities, to structured acquisition of semantic taxonomies from text, and fine-grained semantics such as intensity order. These facets represent structured NLP tasks which have a combinatorially large decision space. Hence, in general, we adopt a structured learning approach, incorporating surface Web-based semantic cues as intuitive features on the full space of decisions. The feature weights are then learned automatically based on a discriminative training approach. Empirically, for each facet, we see significant improvements over the corresponding state-of-the-art.

In the first part of this thesis, we show how Web-based features can be powerful cues to resolving complex syntactic ambiguities. We develop surface n-gram features over the full range of syntactic attachments, encoding both lexical affinities as well as paraphrase-based cues to syntactic structure. These features, when encoded into full-scale, discriminative dependency and constituent parsers, correct a range of error types.

In the next part, we address semantic ambiguities in discourse-level coreference resolution, again using Web n-gram features that capture a range of world knowledge cues to hypernymy, semantic compatibility, and semantic context, as well as general lexical co-occurrence. When added to a state-of-the-art coreference baseline, these Web features provide significant improvements on multiple datasets and metrics.

In the third part, we acquire the semantics itself via structured learning of hypernymy taxonomies. We adopt a probabilistic graphical model formulation which incorporates heterogeneous relational evidence about both hypernymy and siblinghood, captured by surface features based on patterns and statistics from Web n-grams and Wikipedia abstracts. Inference is based on loopy belief propagation and spanning tree algorithms. The system is discriminatively trained on WordNet sub-structures using adaptive subgradient stochastic optimization. On the task of reproducing subhierarchies of WordNet, this approach achieves substantial error reductions over previous work.

Finally, we discuss a fine-grained semantic facet - intensity order, where the relative ranks of near-synonyms such as good, great, and excellent are predicted using Web statistics of phrases like good but not excellent. We employ linear programming to jointly infer the positions on a single scale, such that individual decisions benefit from global information. When ranking English near-synonymous adjectives, this global approach gets substantial improvements over previous work on both pairwise and rank correlation metrics.