Skip to main content
eScholarship
Open Access Publications from the University of California

UC Berkeley

UC Berkeley Electronic Theses and Dissertations bannerUC Berkeley

Designing Assurance Frameworks for Autonomous Systems

Abstract

Autonomous systems that operate in safety-critical domains are in need of assurance frame-works; that is, tools and techniques for preventing failures during deployment and verifying that a system will satisfy specifications. Designing assurance frameworks is especially difficult for learning-enabled systems since machine learning models are not easily interpretable, work in high-dimensional spaces, do not have guarantees on their outputs, and often operate in unstructured and complex environments. The central problem of this dissertation is therefore related to the principles of design for assurance frameworks: what makes a good assurance framework, and how do its components interact with each other and the over-all system? The main challenge is that tools such as uncertainty quantification, conformal prediction, and scenario optimization, while well-studied on their own and promising for producing assurances, are not well-understood in the context of safety and control.In the first part of this dissertation, I study runtime safety filters for modular systems that use uncertainty quantification. I argue that uncertainty quantification, while ill-defined for standalone modules, should be contextualized with respect to how the uncertainty measures are used by the rest of the system. I present novel analysis techniques with which one can compare uncertainty-aware system designs, produce specifications on uncertainty measures, and examine system vulnerabilities, and apply the techniques to real-world systems.In the second part of this dissertation, I examine how to apply statistical verification techniques for control problems. I present three methodologies. In the first, I show how conformal prediction can be used in a safe planning framework in a dynamic environment. In the second, I present a novel technique to perform active learning to calibrate a learned reachable set while simultaneously producing a probabilistic generalization bound. Finally, I introduce a scenario optimization-based hierarchical framework used to guide a deployed controller towards regions that are most likely to be safe. All three methodologies create a bridge between statistics and control, balancing assumptions needed by different tools and bringing to light potentially unintuitive design tradeoffs.