These days, robots are contributing to many aspects of our lives and this role is growing. One of the fundamental problems in robotics is shape representation and mapping, that is necessary for most robotic applications. In this regard, from a stream of lidar scans or RGB-D images we model an object shape or map an environment. One of the main challenges in this regard is an appropriate shape representation with appropriate ray-tracing speed and accuracy. There are many other challenges for mapping like being 3D, continuous, probabilistic, online, containing semantic information, and the possibility of the mapping algorithm to be distributed. In this thesis we propose a novel mapping algorithm based on existing shape representation signed distance function and Gaussian Processes which is able to reconstruct an online, continuous, probabilistic 3-D representations of the geometric surfaces and semantic classes in the environments. Then we extend it to a distributed mapping algorithm that is able to be perform in a network of robots. Additionally, we propose a novel shape representation signed directional distance function (SDDF) that measures signed distance from a query location to a set surface in a desired viewing direction. The main advantage of SDDF, compared to existing shape representations, is that ray tracing can be performed as a look-up operation through a single function call. Additionally, we propose a deep neural network model for SDDF shape learning. A key property of our DeepSDDF model is that it encodes by construction that distance to the surface decreases linearly along the viewing direction. This ensures that the model represents only valid SDDF functions and allows reduction in the input dimensionality and analytical confidence in the distance prediction accuracy even far away from the surface. We show that our DeepSDDF model can represent entire object categories and interpolate or complete shapes from partial views.

This dissertation considers the problem of safe navigation for autonomous mobile robots working in partially known and unknown environments with static and non-adversarial moving obstacles. Given a geometric path generated by standard path planner, we develop reference-governor based tracking control policy to continuous generate proper set-points along the path for downstream low-level stabilizing controller.

This new method systematically puts planning, motion prediction and safety metric design together to achieve environmentally adaptive and safe navigation. Our algorithm balances optimality in travel distance and safety regarding passing clearance. Robots adapt progress speed adaptively according to the sensed environment, being fast in wide open areas and slowdown in narrow passages and taking necessary maneuvers to avoid dangerous incoming obstacles. Directional distance measure, motion prediction and custom costmap are integrated properly to evaluate system risk accurately with respect to local geometry of surrounding environments. Using such risk estimation, reference governor technique and control barrier function are worked together to enable adaptive and safe path tracking in dynamical environments. We validate our algorithm extensively both in simulations and hardware platforms in challenging real-world environments.

Neural implicit surface representations are a promising new development in surfacemodeling. However, the challenges inherent in training neural networks in a continual fashion are still holding them back from being widely used in real-time, incremental scene mapping. We propose a method for learning a neural representation of a signed distance function from trajectories of posed depth images that is both computationally efficient and avoids the problem of catastrophic forgetting. We demonstrate our approach by producing high-quality scene reconstructions in 2D and 3D and incrementally building 2D neural-implicit maps.

Tropical rainforests worldwide are negatively impacted from a variety of human-caused threats. Unfortunately, our ability to study these rainforests is impeded by logistical problems such as their physical inaccessibility, expensive aerial imagery, and/or coarse satellite data. One solution is the use of low-cost, Unmanned Aerial Vehicles (UAV), commonly referred to as drones. Drones are now widely recognized as a tool for ecology, environmental science, and conservation, collecting imagery that is superior to satellite data in resolution. We asked: Can we take advantage of the sub-meter, high-resolution imagery to detect specific tree species or groups, and use these data as indicators of rainforest functional traits and characteristics? We demonstrate a low-cost method for obtaining high-resolution aerial imagery in a rainforest of Belize using a drone over three sites in two rainforest protected areas. We built a workflow that uses Structure from Motion (SfM) on the drone images to create a large orthomosaic and a Deep Convolutional Neural Network (CNN) to classify indicator tree species. We selected: 1) Cohune Palm (Attalea cohune) as they are indicative of past disturbance and current soil condition; and, 2) the dry-season deciduous tree group since deciduousness is an important ecological factor of rainforest structure and function. This framework serves as a guide for tackling difficult ecological challenges and we show two additionally examples of how a similar architecture can help count wildlife populations in the Arctic.

This thesis considers the problem of safe navigation for autonomous mobile robots working in partially known environments with static and non-adversarial dynamic obstacles. A virtual reference governor system is designed to serve as a local regulation point for the real robot system. We calculate a safe set of the robot-governor system by actively measuring the distance from surrounding static obstacles. The motion of the governor is designed to guide the robot towards the goal, while remaining within the safe set. To avoid dynamic obstacles, a control barrier function is built to further constrain the governor-robot system away from dynamic obstacles. A quadratically constrained quadratic program (QCQP) is formulated to minimally modify the governor control input to ensure dynamic obstacle avoidance. Combining these two techniques, the robot can navigate autonomously in unknown environments -- slowing down when approaching obstacles, speeding up in free space, and reacting to the position and velocity of the surrounding dynamic obstacles. Our techniques are demonstrated in simulations and real-world experiments using a ground robot equipped with LiDAR to navigate among a cluttered environment in the presence of humans.

Autonomous robots have the potential to play a critical role in various aspects of modern life, including search and rescue, autonomous driving, medical surgery, agricultural farms, etc. Reinforcement learning algorithms allow intelligent agents to discover optimal behavior through trial and error from the interactions with the environment and have been successfully applied to playing video games, mastering the game of Go and training large language models. In robotics, this data driven learning approach is also promising for locomotion, manipulation and navigation.

When demonstrations are available, an agent can learn to perform a task by imitating expert behavior. However, the agent has to generalize to novel scenarios that are not seen in training. This thesis introduces two aspects to learn generalizable policies from demonstrations. The first method infers a cost function from semantic and geometric information from observations and can generalize to unseen, dynamic, partially observable simulated environments for autonomous driving scenarios. The second method infers task logic from demonstrations which are in turn used as constraints for motion planning. It exploits the hierarchical logic structure from demonstrated trajectories and can generalize to sequential, compositional planning problems.

Another challenge towards deploying robots in the natural world is the ability to bridge the simulation to reality gap. While simulation provides training data at low cost, the policy should be able to account for mismatches in sensing and actuation when deployed on real robots. To address these challenges, this dissertation introduces a latent space alignment approach where policies trained on a source robot can be adapted to a target robot of different embodiments. Finally, this dissertation also presents a sim-to-real method for throwing and catching objects with bimanual robots, where they need to cooperate precisely to interact with diverse objects at high speed.

Recent advances in the field of neural implicit surface representation has led to the use of signed distance functions as a continuous representation of complex surfaces. Though signed distance functions allow for a very simple and concise mapping of the environment, rendering these surfaces from signed distance functions are non-trivial and require iterative methods such as ray marching, leading to high render times. The inclusion of direction in these functions have been proposed and applied to various object-level applications, but have yet to be proposed for scene-level applications. This thesis explores the idea of scene-level directional distance functions and some of the problems that it faces. It then proposes three different methods through which the directional distance function can be implemented and potential solutions to some of the drawbacks of the directional distance function.

We consider the model-based reinforcement learning framework where we are interested in learning a model and control policy for a given objective. We consider modeling the dynamics of an environment using Gaussian Processes or a Bayesian neural network. For Bayesian neural networks we must define how to estimate uncertainty through a neural network and propagate distributions in time. Once we have a continuous model we can apply standard optimal control techniques to learn a policy. We consider the policy to be a radial basis policy and compare it's performance given the different models on a pendulum environment.

The ability for a mobile robot to autonomously and efficiently build a map of an environment, termed robotic exploration, is long sought after by researchers and industry professionals. Existing work in this field focuses on local continuous-space optimization or global discrete-space search over hand-designed graph structures with information-theoretic objectives. Global optimization is critical in practice to achieve a precise trade-off between exploration of new areas and map refinement in observed areas. Many existing approaches solve the optimization with a gradient-based method with poor convergence properties, or by heavily constraining the optimization to a lower dimensional search space. A major contribution of this thesis is the automated construction of a graph of potential robot trajectories, optimized to capture long-horizon maximization of the map's information content. The graph represents a significantly larger space of promising robot trajectories compared to existing motion primitive approaches, leading to a combined improvement in exploration speed and uncertainty reduction. This work also provides a thorough evaluation and comparison of several state-of-the-art autonomous exploration techniques over a large dataset of simulated environments as well as several physical robot trials in cluttered environments.

Semantic understanding and reconstruction of the surrounding 3D environment is a necessary requirement for intelligent robots to autonomously fulfill various tasks like environment exploration, surveillance, autonomous driving, indoor household and healthcare service, to name a few. Although the progress on semantic understanding of 2D images is impressive, building a 3D consistent, meaningful yet compact and scalable semantic map for robotics applications is still challenging. In this dissertation, we develop 3D semantic map approaches for robots in different form, including the dense semantic map and the object-level semantic map. We propose TerrainMesh, a dense semantic map in the form of 3D mesh, for the terrain reconstruction from the aerial images and the sparse depth measurements. The joint 2D-3D and geometric-semantic learning framework is proposed to reconstruct the local semantic meshes and the global semantic mesh can be constructed by merging the local meshes with the help of the SLAM algorithm. We investigate the object-level semantic map constructed from 3D measurements. We propose CORSAIR, a retrieval and registration algorithm for point cloud objects. A 3D sparse convolution neural network model is trained to extract the global features for similar shape retrieval and the local per-point features for correspondence generation for pose registration with the help of symmetry. We develop ELLIPSDF, a bi-level object shape model including a coarse level of 3D ellipsoid and a fine level of 3D continuous signed distance function (SDF). We also design the approach to initialize and optimize the object pose together with the bi-level shape from the multiple depth image observations. We also propose the object-level semantic map from the 2D images and investigate its connections with the localization task. We introduce the object mesh model and its observation model of semantic instance segmentation and semantic keypoints. We derive the observation residual function and minimize it to optimize both the object states and the camera poses. We develop OrcVIO, object residual constrained visual-inertial odometry with object ellipsoid and semantic keypoints model. We implement the observation residual model between the ellipsoid and its 2D semantic bounding box and semantic keypoints and connect this with MSCKF framework to implement the online tightly coupled estimation of object and IMU-camera states. The object-level semantic map also provides a meaningful yet efficient representation of the environment. Finally we discuss the potential directions to further extend the 3D semantic understanding technique for robotics.