UC Berkeley

Despite the exciting progress in computer vision in the field of autonomous driving, understanding efficiently which cues or objects are the most crucial ones in a crowded traffic scene is still a big challenge. Human drivers can quickly identify the important visual cues or objects in their blurry periphery vision and then make eye movements to direct their more accurate foveal vision to the important regions. Therefore, driver eye movements may be what computer vision can borrow from human vision to make autonomous driving systems better at locating and understanding the important regions of crowded traffic scenes. Meanwhile, the large-scale datasets and advanced object recognition algorithms that emerged in the field of autonomous driving make it possible to study classical human vision science problems in natural driving situations.

Here, we used driver eye movements to improve autonomous driving models and studied visual crowding—the bottleneck of human object recognition—in realistic driving situations through driver eye movements. First, we developed a new protocol that collects driver eye movements in an offline manner for large-scale driving video datasets. We built a deep neural network that predicts human driver gaze from dash camera videos for various driving scenarios. Our model outperformed the current state-of-the-art model. Furthermore, we incorporated the driver gaze prediction model into an autonomous driving model to make a new periphery-fovea multi-resolution driving model that predicts vehicle speed from dash camera videos. This model combines low-resolution input of the whole video frames and high-resolution input from predicted gaze locations to predict vehicle speed. We show that the added human gaze significantly improves the driving accuracy and that our periphery-fovea multi-resolution model outperforms a uni-resolution periphery-only model that has the same amount of floating-point operations. Finally, we studied visual crowding in driving situations. We show that crowding occurs in natural driving scenes and that the degree of crowding correlates with altered saccade localization in realistic driving-like situations. Together, these studies demonstrate the application of driver eye movements in making safer and more efficient autonomous driving models and show strong evidence of visual crowding in driving situations via the analysis of driver eye movements. These studies also present examples of combining human vision and computer vision to get mutual benefits from both fields.