Robots are becoming increasingly weaved into the fabric of our society, from self-driving cars on our streets to assistive manipulators in our homes. To act in the world, robots rely on a representation of salient features of the task: for example, to hand me a cup of coffee, the robot considers movement efficiency and cup orientation in its behavior. However, if we want robots to act for and with people, their representations must not be just functional but also reflective of what humans care about, i.e. their representations must be aligned with humans'. What's holding us back from successful human-robot interaction is that these representations are often misaligned, resulting in anything from miscoordination and misunderstandings, to learning and executing dangerous behaviors.

To learn the human's representation of what matters in a task, typical methods rely on data sets of human behavior but this data cannot reflect every individual, environment, and task the robot will be exposed to. This dissertation advocates that we should instead treat humans as active participants in the interaction not as static data sources: robots must engage with humans in an interactive process for finding a shared representation. We formalize the representation alignment problem as a joint search for a common representation. Then, rather than hoping that representations will naturally be aligned, we propose having humans directly teach them to robots with representation-specific input. Next, we enable robots to automatically detect representation misalignment with the human by estimating a confidence over how much the robot's representation can explain the human's behavior. We demonstrate how human-aligned representations can lead to novel human behavior models with broad implications beyond robotics, to econometrics and cognitive science. Finally, this thesis concludes by asking ``How can robots help the human-robot team converge to a shared representation?'' and discusses opportunities for future work in expanding representation alignment for seamless human-robot interaction.

From autonomous cars to systems operating in people's homes, robots must interact with humans. What makes this hard is that human behavior--especially when interacting with other agents--is vastly complex, varying between individuals, environments, and over time. A modern approach to deal with this problem is to rely on data and machine learning throughout the design process and deployment to build and refine models of humans. However, by blindly trusting their data-driven human models, robots might confidently plan unsafe behaviors around people, resulting in anything from miscoordination to potentially even dangerous collisions.

This dissertation aims to lay the foundations for formalizing and ensuring safety in human-robot interaction, particularly when robots learn from and about people. It discusses how treating robot learning algorithms as dynamical systems driven by human data enables safe human-robot interaction. We first introduce a Bayesian monitor which infers online if the robot's learned human model can evolve to well-explain observed human data. We then discuss how a novel, control-theoretic problem formulation enables us to formally quantify what the robot could learn online from human data and how quickly this learning could be achieved. Coupling these ideas with robot motion planning algorithms, we demonstrate how robots can safely and automatically adapt their behavior based on how trustworthy their learned human models are. This thesis ends by taking a step back and raising the question: "What is the ‘right’ notion of safety when robots interact with people?'' and discusses opportunities for how rethinking our notions of safety can capture more subtle aspects of human-robot interaction.

Deploying learning systems in the real-world requires aligning their objectives with those of the humans they interact with. Existing algorithmic approaches for this alignment try to infer these objectives through human feedback. The correctness of these algorithms crucially depends on several simplifying assumptions on 1) how humans represent these objectives, 2) how humans respond to queries given these objectives, and 3) how well the hypothesis space represents these objectives. In this thesis, we question the robustness of existing approaches to misspecifications in these assumptions and develop principled approaches to overcome such misspecifications.

We begin by studying misspecifications in the hypothesis class assumed by the learner and propose an agnostic learning setup where we demonstrate that all existing approaches based on learning from comparisons would incur constant regret. We further show that it is necessary for humans to provide more detailed feedback in the form of higher-order comparisons and obtain sharp bounds on the regret as a function of the order of comparisons. Next, we focus on misspecifications in human behavioral models and establish, through both theoretical and empirical analyses, that inverse RL methods can be extremely brittle in worst case. However, under reasonable assumptions, we exhibit that these methods do exhibit robustness and are able to recover underlying reward functions up to a small error term. We then proceed to study misspecifications in assumptions on how humans represent objective functions. We begin by showing that taking a uni-criterion approach to modeling human preferences fails to capture real-world human objectives and propose a new multi-criteria comparison based framework which overcome these limitations. In the next part, we shift our focus to hand-specified reward functions in reinforcement learning, an alternative to learning rewards from humans. We empirically study the effects of such misspecifications showing that over-optimizing such proxy rewards can hurt performance in the long run.

One of the outstanding challenges in the field of human-computer interaction is building assistive interfaces that help users with the perception and control of complex systems, such as cars, quadcopters, and prosthetic limbs. In this thesis, we propose machine learning algorithms for automatically designing personalized, adaptive interfaces that improve users' performance on sequential decision-making tasks. First, we present work that uses theory of mind to model irrational user behavior as rational with respect to incorrect internal beliefs about how the world works, and leverages this assumption to assist users by modifying their observations and actions. Second, we present work that uses model-free reinforcement learning from human feedback to fine-tune user actions, with minimal assumptions about user behavior. We demonstrate the effectiveness of our methods through experiments with human participants, in which users play the Lunar Lander video game, perform simulated navigation tasks, and land a quadcopter.

Typically when learning about what people want and don't want, we look to human action as evidence: what reward they specify, how they perform a task, or what preferences they express can all provide useful information about what an agent should do. This is essential in order to build AI systems that do what we intend them to do. However, existing methods require a lot of expensive human feedback in order to learn even simple tasks. This dissertation argues that there is an additional source of information that is rather helpful: the state of the world.

The key insight of this dissertation is that when a robot is deployed in an environment that humans have been acting in, the state of the environment is already optimized for what humans want, and is thus informative about human preferences.

We formalize this setting by assuming that a human H has been acting in an environment for some time, and a robot R observes the final state produced. From this final state, R must infer as much as possible about H's reward function. We analyze this problem formulation theoretically and show that it is particularly well suited to inferring aspects of the state that should not be changed -- exactly the aspects of the reward that H is likely to forget to specify. We develop an algorithm using dynamic programming for tabular environments, analogously to value iteration, and demonstrate its behavior on several simple environments. To scale to high-dimensional environments, we use function approximators judiciously to allow the various parts of our algorithm to be trained without needing to enumerate all possible states.

Of course, there is no point in learning about H's reward function unless we use it to guide R's decision-making. While we could have R simply optimize the inferred reward, this suffers from a "status quo bias": the inferred reward is likely to strongly prefer the observed state, since by assumption it is already optimized for H's preferences. To get R to make changes to the environment, we will usually need to integrate the inferred reward with other sources of preference information. In order to support such reward combination, we use a model in which R must maximize an unknown reward function known only to H. Learning from the state of the world arises as an instrumentally useful behavior in such a setting, and can serve to form a prior belief over the reward function that can then be updated after further interaction with H.

Specifying the correct objective function for a machine learning system is often difficult and error-prone. This thesis focuses on learning objective functions from many kinds of human inputs. It is comprised of three parts. First, we present our formalism, reward-rational choice (RRC), that unifies reward learning from many diverse signals. The key insight is that human behavior can often be modeled as a reward-rational implicit choice – a choice from an implicit set of options, that is approximately rational for the intended reward. We show how much of the prior work, despite using many diverse modalities of feedback, can be seen as an instantiation of RRC.

In the second part, we discuss implications of the RRC formalism. In particular, it allows us to learn from multiple feedback types at once. Through case studies and experiments, we show how RRC can be used to combine and actively select from feedback types. Furthermore, once a person has access to multiple types of feedback, even their choice of feedback type itself provides information to learn the reward function from. We use RRC to formalize and learn from a person's meta-choice, the choice of feedback type itself.

Finally, in the third part, we study settings in which the human may violate the reward-rational assumption. First, we consider the case where the human may be pedagogic, i.e., optimizing for teaching the reward function. We show that the reward-rational assumption provides robust reward inference even when the human is pedagogic. Second, we consider the case where the human may face temptation and act in ways that systematically deviate from their target preferences. We theoretically analyze such a setting and show that, with the right feedback type, one can still efficiently recover the individual's preferences. Lastly, we consider the recommender system setting. There, it is difficult to model all user behaviors as rational, but by leveraging one strong, explicit signal (e.g. ``don't show me this''), we are still able to operationalize and optimize for a notion of ``value'' on these systems.

As robots become more capable and commonplace, it becomes increasingly important that they are transparent to humans. People need to have accurate mental models of a robot, so that they can anticipate what it will do, know when and where not to rely it, and understand why it failed. This helps engineers ensure safety and robustness of the robot systems they develop, and enables human end-users to interact more safely and seamlessly with robots.

This thesis introduces a framework for producing robot behavior that increases transparency. Our key insight is that a robot's actions do not just influence the physical world; they also inevitably influence a human observer's mental model of the robot. We attempt to model the latter---how humans might make inferences about a robot's objectives, policy, and capabilities from observations of its behavior---so that we can then present examples of robot behavior that optimally bring the human's understanding closer to the true robot model. In this way, our framework casts transparency as an optimization problem.

Part I introduces our framework of optimizing for robot transparency, and applies it in three ways: communicating a robot's objectives, which situations it can handle, and why it is incapable of performing a task. Part II investigates how transparency is useful not just for safe and seamless interaction, but also for learning. When humans teach a robot, giving human teachers transparency regarding what the robot has learned so far makes it easier for them to select informative teaching examples.

The field of artificial intelligence has seen serious progress in recent years, and has also caused serious concerns that range from the immediate harms caused by systems that replicate harmful biases to the more distant worry that effective goal-directed systems may, at a certain level of performance, be able to subvert meaningful control efforts. In this dissertation, I argue the following thesis: 1. The use of incomplete or incorrect incentives to specify the target behavior for an autonomous system creates a value alignment problem between the principal(s), on whose behalf a system acts, and the system itself; 2. This value alignment problem can be approached in theory and practice through the development of systems that are responsive to uncertainty about the principal’s true, unobserved, intended goal; and 3. Value alignment problems can be modeled as a class of cooperative assistance games, which are computationally similar to the class of partially-observed Markov decision processes. This model captures the principal’s capacity to behave strategically in coordination with the autonomous system. It leads to distinct solutions to alignment problems, compared with more traditional approaches to preference learning like inverse reinforcement learning, and demonstrates the need for strategically robust alignment solutions.

Chapter 2 goes over background knowledge needed for the work. Chapter 3 argues the first part of the thesis. First, in Section 3.1 we consider an order-following problem between a robot and a human. We show that improving on the human player’s performance requires that the robot deviate from the human’s orders. However, if the robot has an incomplete preference model (i.e., it fails to model properties of the world that the person cares about), then there is persistent misalignment in the sense that the robot takes suboptimal actions with positive probability indefinitely. Then, in Section 3.2, we consider the problem of optimizing an incomplete proxy metric and show that this phenomenon is a consequence of incompleteness and shared resources. That is, we provide general conditions under which optimizing any fixed incomplete representation of preferences will lead to arbitrarily large losses of utility for the human player. We identify dynamic incentive protocols and impact minimization as theoretical solutions to this problem.

Next, Chapter 4 deals with the second part of the thesis. We first show, in Section 4.1, that uncertainty about utility evaluations creates incentives to get supervision from the human player. Then, in Section 4.2 and Section 4.3, we demonstrate how to use uncertainty about utility evaluations to implement reward learning approaches that penalize negative side-effects and support dynamic incentive protocols. Specifically, we show how to apply Bayesian inference to learn a distribution over potential true utility functions, given the observation of a proxy in a specific development context.

Chapter 5 deals with the third part of the thesis. We introduce cooperative inverse reinforcement learning (CIRL), which formalizes the base case of assistance games. CIRL models dyadic value alignment between a human principal H and a robot assistant R. This game-theoretic framework models H’s incentive to be pedagogic. We show that pedagogical solutions to value alignment can be substantially more efficient than methods based on, e.g., imitation learning. Additionally, we provide theoretical results that support a family of efficient algorithms for CIRL that adapt standard approaches for solving POMDPs to compute pedagogical equilibria.

Finally, Chapter 6 considers the final component of the thesis, the need for robust solutions that can handle strategy variation on the part of H. We introduce a setting where R assists H in solving a multi-armed bandit. As in Section 3.1, H’s actions tell R which of the k different arms to pull. However, this introduces the complication that H does not know which arm is optimal a priori. We show that this setting admits efficient strategies where H treats their actions as purely communicative. These communication solutions can achieve optimal learning performance, but perform arbitrarily poorly if the encoding strategy used by H is misaligned with R’s decoding strategy.

We conclude with a discussion of related work in Chapter 7 and proposals for future work in Chapter 8.

Many modern robotic systems rely on models to complete their tasks. A model captures all aspects of the environment relevant to the robot, and enables the robot to predict the result of any action it may take. Such a model enables the robot to plan and execute an efficient behavior to complete a given task.

Unfortunately, however, no model is perfect. Many real-world phenomena that a robot may encounter --- from contact forces such as friction, to unpredictable human behavior --- are exceedingly difficult to model precisely. Instead, we typically make certain simplifying assumptions, which make it easier to build a model of the environment for the robot to use in planning and execution. These assumptions, however, invariably break down somewhere. Since the robot relies on such simplified models for planning and execution, its performance can suffer as a result of these inaccuracies. We refer to such models as being unreliable, since the robot's planning and execution systems are unable to rely on the model's predictions to produce a behavior to ensure that the robot completes its task in the real-world with a good level of performance. It is therefore critical to design the robot's planning and execution systems to be as robust as possible to unreliable models.

To that end, we focus on increasing the robustness of two key aspects of a model-based planning and execution system: first, how to choose a model that best captures the robot's current environment; and, second, how to plan with that model, despite the possibility that it is unreliable. In this dissertation, we introduce three new algorithms towards increasing the robustness of a robot's planning and execution system, to unreliable models of the real-world. First, we introduce an approach to enable the robot to choose the best model for its current situation online, from a larger set of possible models. However, even this best model may not be perfectly reliable. To address this, we complement our approach with the ability to reason about where the robot should and should not rely on the model, which we describe in the second part of this dissertation. In doing so, the robot is able to leverage the model where it is accurate, but avoid regions where the model may lead the robot's plan astray. More specifically, we present two implementations of this idea in the second part of this dissertation, the main difference being in the type of model that each approach assumes --- the first focuses on deterministic (or, certain) models, whereas the second focuses on probabilistic (or, uncertain) models.

Throughout this dissertation, to evaluate the effectiveness of our approaches, we present a set of experiments in simulation, as well as on various real-robots. These robots include a 7 degree-of-freedom robot arm, a non-holonomic ground robot, and a free-flying space robot that operates in zero-gravity. Overall, we show how our approaches improve the robot's ability to plan and execute behaviors with greater robustness to unreliable models. Finally, we summarize several preliminary ideas and future research directions around building a unified framework for model-based planning and execution with unreliable models, bringing together all of the key contributions in this dissertation.