Skip to main content
Open Access Publications from the University of California

UC Santa Cruz

UC Santa Cruz Electronic Theses and Dissertations bannerUC Santa Cruz

Tightly-Coupled LiDAR and Camera for Autonomous Vehicles

Creative Commons 'BY-NC' version 4.0 license

Autonomous driving has received remarkable attention and investment from both industry and academia over the past couple of years. Researchers from different fields are contributing to solve various challenges towards fully autonomous cars, e.g. optics, light sensing, deep learning, statistics.

A driver-less car, first and foremost, must be equipped with the right choice of complimentary sensors in order to measure the surroundings. Then it needs to process the measurements to understand the scene, perform path planning, and react to the environment. In this work, we focus on processing raw sensor measurements to extract meaningful physical or geometrical features like segments, velocity, depth, graphical relations between different segments, etc.

In this dissertation, our primary focus is on leveraging the complimentary features of two automotive grade sensors: LiDAR and Camera. The goal is to break the scene into independently moving segments, reconstruct their surface, find inter-segment connections, and estimate their three-dimensional velocity. As the main contribution of this work, we combine measurements in LiDAR points and Camera pixels at the raw level. This is opposed to tracking-by-detection or high-level fusion where sensors are fused after object detection. We lay a graph-based approach to generate a set of three-dimensional superpixels (supersurfaces) that cover the whole environment. These supersurfaces are optimized according to connectivity features in both LiDAR and Camera space. At a higher level, the connections between neighboring supersurfaces are also modeled and considered in subsequent modules. We also develop an energy function that explicitly penalizes three-dimensional velocity vectors based on both image pixels (optical flow term) and LiDAR points (geometric distance term). This energy is further regularized to maintain edge-preserving temporal and spatial smoothness. This leads to tightly-coupled sensor fusion, and benefits from more robustness in velocity estimation.

One important feature of this work is its independence from object detection, unlike many methods that rely on supervised learning to detect object instances. This feature makes our algorithm capable of handling unknown classes of objects. Another key characteristic of our fusion approach is the notion of adaptive \emph{processing resolution}. Objects, depending on their distance, are processed at different pyramid levels (and different resolutions) which eliminates redundant computations and still maintains desired estimation error bounds.

Our prototyped experimental results show an improvement in velocity estimation error with respect to the state of the art. We also demonstrate the functionality of the overall algorithm by assembling all submodules into an efficient CUDA implementation. Several changes and adaptations are applied to make the algorithm optimized for running on GPU.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View