Advancements in deep learning have catalyzed growth in robotic applications, extending their utility beyond constrained settings. Nevertheless, a significant challenge remains in en- abling robots to efficiently navigate and interact within unstructured and dynamic environments. Existing methods in robot navigation require the use of dense geometric representations such as high definition maps or full 3D reconstruction. But these methods are non trivial and consume significant resources in both creation and usage. They also become stale for environments with constant changes. To be able to scale in terms of size and novelty of the environment, new algorithms that use representations accounting for semantics is required. Besides that, to be able to interact and collaborate with humans, these representations need to be able to ground the visual information in other modalities such as text, while retaining a long term memory. This thesis presents work on these directions, development of new approach and the discussion on the experiments applying these methods to navigation and robot instruction following in home environments.

Current service robots perform flawless demos under close human supervision, but they often fail when working autonomously in long-term deployments. This dissertation studies the failures in long-term autonomous service robots and proposes methods to improve their reliability.

We built TritonBot, a receptionist and tour-guide robot, as a realistic example to discover the failure modes of a long-term autonomous service robot. TritonBot recognizes people's faces, talks to people, and shows people the labs and facilities in a university building. We deployed TritonBot for hundreds of hours to identify failure modes and common issues on service robots.

Following the experience from TritonBot, we designed two reliability engineering methods to improve the robustness of service robots. First, we found software encapsulation and dynamic orchestration streamline development workflows and avoid resource contention in service robots. Software encapsulation allows the developers to pack software into self-contained containers and simplify development workflows, and dynamic orchestration schedules the components on demand to avoid CPU/memory resource contention. We developed Rorg, a Linux container-based scheme to manage software components on service robots. Second, we found simulating a broad spectrum of rare failures at system level exposes design flaws and improves the robustness of service robots. Design errors in robotics are challenging to discover due to the need for extensive and resource-demanding testing. Broad-spectrum system-level failure injection exposes both software- and hardware-related design flaws and assists developers in reproducing rare failures, verifying the fixes, and testing the robustness of a robot system. We implemented RoboVac, an extensible and convenient fault injection framework that works at the system level and covers many failure patterns seen in long-term autonomous service robot deployments.

After working with TritonBot for two years and implementing reliability engineering methods, we concluded a few design principles for a long-term autonomous service robot at different levels in the system hierarchy. These design principles guide robust and reliable long-term autonomous service robot designs.

We use a set of automated tools, engineering methods, and design principles to build service robots that are available 24x7, and we call it "Reliability Engineering for Long-term Deployment of Autonomous Service Robots."

Human-like manipulation for robots has been an ongoing research effort for decades. Among such efforts, stable pick-and-place is one of the simplest yet impactful tasks. The ability to perform stable pick-and-place tasks, especially with unseen object in cluttered environments, would enable robots to handle common household object and eventually perform more complex tasks in kitchens, labs and offices. This thesis presents a novel grasp analyzer to identify stable sliding and lifting grasp poses for parallel grippers. This grasp analyzer combines learning-based models with physics-based models to ensure performance and robustness. Combined with a multi-modal planner, the grasp analyzer was proved with experiments to work both in simulation and real world.

Robotic grasping in a complex environment is one of the fundamental challenges for home-assistant robots. Complex environment grasping has been extensively studied in industrial bin-picking scenarios, where reliably grasping objects from unorganized heaps is challenging due to sensor noise, obstructions, and occlusions. However, bin picking is still relatively easier than grasping common household objects from a structured clutter in a home environment because the robot cannot knock over neighboring objects during the grasping motion. Recently, there have been several attempts to tackle the grasping-in-structured-clutter problem. In our experiments, we found these methods either hard to adapt to our simulated environment without extra tuning or generate too few stable grasps to successfully grasp the objects. The overviews and detailed analyses of these existing grasping approaches will appear in the first half of this thesis.

In the second half of this thesis, we investigate the idea of using a physical simulator as an intermediate step to generate a grasp trajectory proposal. At a high level, we propose a two-step approach to solve the grasping-in-structured-clutter problem. First, we collect RGB-D observations to reconstruct the environment in a physical simulator via 9 degree-of-freedom (DoF) category-level object pose estimation, CAD model matching, and physical support refinement. Then, we perform antipodal grasp sampling, collision-free motion planning, and grasp execution in the simulator and directly transfer the robot arm’s motion trajectory to the original environment.

To generate a 9-DoF category-level object pose estimate, we extend a state-of-the-art 6-DoF instance-level object pose estimation network. In our experiments, we found the 9-DoF pose estimation network can reach performance comparable to the state-of-the-art on a category-level object pose estimation dataset. Relying on only the top-down view of the environment, we reconstructed the environment using the proposed two-step approach and evaluated the grasp transfer success. The results show further room for improvements in the model matching process. Future directions and some ideas will be discussed towards the end of this thesis.

We hope the work of scene reconstruction for simulated grasp search and trajectory transfer will help future research of robotics manipulation in complex environments.

Recent advancements in statistical learning and computational abilities have enabled autonomous vehicle technology to develop at a much faster rate. While many self-driving architectures previously introduced are capable of operating under highly dynamic environments, many of these are constrained to smaller-scale deployments, heavy dependence on high-definition (HD) maps that require constant maintenance and tedious manual labeling. This thesis is trying to address the mapping problem in the context of autonomous vehicle applications. We study the existing map representations in the literature and analyze the performance of parametric mapping algorithms in both indoor and outdoor environments. We then propose to fuse image and pre-built point cloud map information to perform automatic and accurate labeling of static landmarks such as roads, sidewalks, crosswalks, and lanes. The method performs semantic segmentation on 2D images, associates the semantic labels with point cloud maps to accurately localize them in the world, and leverages the confusion matrix formulation to construct a probabilistic semantic map in bird's eye view from semantic point clouds. Experiments from data collected in an urban environment show that this model is able to predict most road features and can be extended for automatically incorporating road features into HD maps with potential future work directions.

Despite the latest breakthroughs and technological advancements in state of the art autonomous vehicle systems, many of these systems still experience challenges in a variety of realistic scenarios and environments that prevent them from generalizing to new situations. In this study, two development platforms are designed, tested, deployed and benchmarked to analyze the implications of micro-transit applications and the autonomy needed for fully autonomous systems.

Security and usability are often paradoxically portrayed as zero-sum trade-offs; this is especially true of cyber-physical systems. Historically, security has taken a back seat in open source robotic software initiatives, such as Robot Operating System (ROS), which have focused on development productivity instead of deployment hardening, and this has contributed, perhaps inadvertently, to the belated adoption of ROS by industry, impeding technical transfer between industry and academia. This dissertation studies methods for automating and verifying middleware configurations to improve the security for distributed robotic systems.

We introduce procedural tooling for automating the setup of authenticated encryption and access control across the distributed data bus network used in robotic middleware frameworks, in a concerted effort to remove sources of errors such as human factors during configuration development. This includes an immutable logging framework with which to record event data used by any tooling in order to continuously profile and/or monitor system security. We then present methods for conducting passive reachability and controllability analysis over the data bus, and for combining that with static information flow control to guarantee isolation between sensitive or mission critical computations over larger integrations. Finally, we propose principles for the design, verification, and deployment of robotic architectures composed of partially distrusting components. We seek to establish a set of automated tools, methods, and design principles to compose and audit multiple robotic subsystems into hardened stacks for service and industry, and we call it "Usable Security and Verification for Distributed Robotic Systems."

Tool use and improvisational tool use is a hallmark of human (and animal) intelligence. When an appropriate tool for a task is not available, we can innovate and design a new tool, or use an existing object in a non-canonical way to accomplish the task, by reasoning about the underlying nature of the task. Despite the impressive capabilities of robots to learn narrow yet complex tasks using tools, innovative tool use by robots still remains an open and a significant challenge. We seek to address this challenge in the context of causal affordance based planning for the robot to reason about the constraints of the task, and thereby select the appropriate tool and its usage. This is done through providing semantic annotations to task preconditions in a Planning Domain Definition Language (PDDL) framework, derived from the inherent constraints of the task. Subsequently, we extend a standard planning algorithm to exploit these semantics, and demonstrate its application in improvisational tool use.

To enable a robotic platform that is low-cost, yet useful for manipulation tasks in household and low-volume production settings without prior calibration, two algorithms for grasp viewpoint planning are proposed. Both algorithms rely on performing motions on a virtual sphere in an object-centric workspace, utilizing only RGB images from an eye-in-hand camera to reach an optimal grasping viewpoint. The first method succeeds on grasping more than 80 \% of the time on the Rethink Robotics Baxter Platform by utilizing image moments to calculate an optimal grasp pose, and the second method is capable of orienting the end effector within 5 degrees of an object on a simulated Universal Robotics UR5 platform using a Convolutional Neural Network (CNN). From experiment results of these two algorithms, it can be shown that an RGB camera is sufficient for viewpoint planning and grasping of novel objects with simple geometries. Additionally, a grasping model can be trained for a new object to increase alignment accuracy and grasp success rates.

Service robots are designed to assist and aid humans in the day to day tasks as well as specific and minute tasks. In recent years there has been an increase in both demand and production of service robots. These robots are slowly becoming more prolific both in personal settings such as the home, and professional settings such as restaurants and hospitals. Despite this growth, there are still major challenges preventing service robots from being ubiquitous and operating intelligently in these unstructured domestic environments.

Like many autonomous robots, the technical challenges of service robots revolve around three main facets: sensing and modeling their environment, understanding and reasoning about said environment, and acting in a way that is safe and conducive towards a task goal. Unlike industry robots, service robots operate in dynamic environments and have tasks that cannot always be completed repetitively. This presents a rich set of challenges to solve in all three areas.

One exemplary task of this for service robots is tidying objects within the home. This task is important as it helps assist with activity of daily living (ADL) while meeting consumer demand. A few concrete illustrations of the challenges of this task are shown when we consider the robot must sense objects and ground language of these objects before we can determine where the objects belong. Notions of a correct placement vary as each user has their own personal preference to organization. Grasping and manipulating the objects as well as navigating within the environment must also be accomplished in order to successfully complete the task.

In this dissertation we present several contributions that help address these challenges and the surrounding gaps towards solving these issues. First, we explore the use of a service robot to put away the groceries in a precise and stable configuration that satisfies the desired semantic relationship of grocery and pantry objects. Second we explore the ability of our robot to put away objects throughout a home environment based on a user preference using a recommender system approach. We share details of our implementation of these systems through real-world experiments.