Modern vehicular automation technology is enabled by complex interactive systems composed of discrete blocks that accomplish specific tasks. These blocks are responsible for perceiving the environment surrounding the vehicle, predicting the future states and intents of other agents in its vicinity, planning the vehicle's path with an eventual goal in mind, and finally, providing control signals and actuations that achieve this goal. While such systems are widely adopted, they fail to consider a key component of the driving process - the human occupants inside the vehicle. This dissertation proposes that to ensure a safe, smooth, and enjoyable travel experience, it is essential for the human driver and vehicular automation to be aware of each other's states and impending failures, i.e., a relationship built on earned trust rather than blind faith.

To codify this requirement, we employ the Looking-In & Looking-Out (LILO) approach, where systems simultaneously model the inside and outside of vehicles. In addition to the set of "looking-out" tasks that are used in conventional automation, we propose a parallel set of "looking-in" tasks that model and analyze the interior of the vehicle. The outputs generated by both these parallel systems can then be integrated to provide useful controls based on a more complete understanding of the circumstances and the environment.

This dissertation presents the models and algorithms we have developed to accomplish the individual tasks that make up the system described above. This includes our research on real-time models for tasks such as object detection, multi-object tracking, and maneuver/trajectory prediction. We also introduce new approaches for the relatively under-explored "looking-in" tasks - which we hope will inspire many such studies in the future. Some of our contributions in this space include real-time models to analyze different aspects of the driver's state, data augmentation and automatic labelling schemes for tasks with limited data, and solutions to other common hindrances. Finally, we provide a concrete example of how the LILO framework can be used in practice - where the driver’s state estimated by "looking-in" is used in conjunction with metrics computed by "looking-out" to initiate safer and smoother control transitions.

Behavior analysis of vehicles surrounding the ego-vehicle is an essential component in safe and pleasant autonomous driving. This study develops a framework for activity classification of observed on-road vehicles using 3D trajectory cues and a Long Short Term Memory (LSTM) model. As a case study, we aim to classify maneuvers of surrounding vehicles at four way intersections. A variety of sensros including stereo cameras, LIDAR, GPS, and IMU measurements are used to extract ego-motion compensated surround trajectories from five different datasets. The impact of different prediction label space choices, feature space input, noisy/missing trajectory data, and LSTM model architectures are analyzed, presenting the strengths and limitations of the proposed approach.

Autonomous vehicles operate in highly interactive environments. They share the road with humans and human driven vehicles, and they may share control with humans in the cabin. Consider scenarios such as freeway merges, unsignalized intersections or unprotected turns. These require cooperating with other on-road agents and predicting their intent and future motion. Similarly, consider scenarios in partial or conditional autonomy where control needs to be transferred to a human driver. It is critical to predict the driver's takeover readiness and reaction times to ensure safe transfer of control.

The goal of this dissertation is to develop models for predicting agent behavior in and around autonomous vehicles. We present our contributions in two parts. In part I, we address the task of trajectory prediction of surrounding agents. We propose models that incorporate multi-agent behavior, generalize to novel scene layouts, and output a multimodal distribution over future trajectories. Concretely, our contributions are: (i) A unified framework for maneuver classification and trajectory prediction for highway traffic. (ii) An LSTM encoder-decoder with convolutional social pooling for modeling agent-agent interaction, and maneuver conditioned decoders for predicting a multimodal distribution. (iii) P2T: a model that infers goals and path preferences of agents in novel scenes using a discrete grid-based policy and predicts scene compliant trajectories. (iv) PGP: a model that predicts trajectories conditioned on paths traversed by a discrete policy in a graph representation of the scene, leading to computational efficiency and better accuracy.

In part II we focus on agents inside the autonomous vehicle. We address control transitions where control needs to be transferred from the vehicle to a human driver via a takeover request. Given the driver's gaze, hand and foot activity prior to the takeover request, we predict their readiness and reaction times during takeovers. Our contributions are: (i) a metric for the driver's takeover readiness based purely on observable cues and a model to estimate it, (ii) a model for predicting takeover time and analysis using a real-world dataset of control transitions.

A mobile platform mounted with omnidirectional vision sensor (ODVS) can be used to monitor large areas and detect interesting events such as independently moving persons and vehicles. To avoid false alarms due to extraneous features, the image motion induced by the moving platform should be compensated. This paper describes a formulation and application of parametric egomotion compensation for an ODVS. Omni images give 360. view of surroundings but undergo considerable image distortion. To account for these distortions, the parametric planar motion model is integrated with the transformations into omni image space. Prior knowledge of approximate camera calibration and camera speed is integrated with the estimation process using a Bayesian approach. Iterative, coarse-to-fine, gradient-based estimation is used to correct the motion parameters for vibrations and other inaccuracies in prior knowledge. Experiments with a camera mounted on various types of mobile platforms demonstrate successful detection of moving persons and vehicles.

Autonomous vehicles face significant challenges in understanding pedestrian behavior, particularly in urban environments. The system must recognize pedestrians’ intentions and anticipate their actions to achieve intelligent driving. This paper focuses on predicting pedestrian crossings, aiming to enable oncoming vehicles to react in a timely manner. We investigate the effectiveness of various input modalities for pedestrian crossing prediction, including human poses, bounding boxes and ego vehicle speed features. We propose a novel lightweight architec- ture based on LSTM and attention to accurately identifying crossing pedestrians. Our methods evaluated on two widely used public datasets for pedestrian behavior, PIE and JAAD datasets,and our algorithm achieved a state-of-the-art performance in both datasets.

Autonomous driving systems involved in perception and planning require large volumes of carefully annotated data for learning and validation. These same systems also must be aware of failure cases so that they can safely request and initiate control transitions to human drivers or remote operators. In this dissertation, I present novelty detection as a unifying solution to both of these problems. Through novelty detection, active learning algorithms can reduce annotation costs by intelligently selecting informative data, which I demonstrate on tasks of 3D object detection and vehicle trajectory prediction. Similarly, novelty detection acts as a requisite step for safely handling hazardous scenarios. Lastly, I present the concept of salience as a property of road objects which expresses their criticality to control decisions, discussing the relevance of this property in developing machine learning systems which have stronger learning and validation over safety-critical scene elements for autonomous driving and can adapt to novelty found in the open world.

Modeling spatio-temporal contextual information is fundamental in computer vision, with particular relevance to robotic intelligence and autonomous driving. We develop several frameworks for context modeling in image, video, multi-modal, and multi-cue data with applications to human-robot interactivity, in particular to the domain of intelligent vehicles. With the goal of developing contextual systems for interactivity, several key contributions are proposed: (1) A contextual framework for robust image-level scene understanding, including detection and localization of vehicles, pedestrians, and parts of humans (e.g. hands) in on-road setting, (2) A spatio-temporal, multi-modal, and multi-cue model which reasons over the complex interplay between the human (hand, head, and foot coordination), vehicle (speed, yaw-rate, etc.), and surround spatio-temporal context (agents, scene information) cues for understanding behavior and predicting activities, (3) A human-centric framework for object recognition and visual scene analysis, developed by studying a notion of object importance and relevance as measured in a spatio-temporal context of navigating a vehicle. The final contribution unifies the aforementioned components of the thesis, including spatio-temporal object recognition, human perception modeling, and behavior and intent prediction into a single research task. Although the data and case studies in this work emphasize the safety-critical settings of navigating a vehicle, the contributions of this thesis are general and can therefore be applied to a wider array of applications involving human-machine interactivity.

This dissertation seeks to enable intelligent vehicles to see, to predict intentions, to understand and to model the state of driver.

We developed a state of the art vision based non-contact gaze estimation framework by carefully designing submodules which will build up to achieve continuous and robust estimation. Key modules in this system include, face detection using deep convolutional neural networks, landmark estimation from cascaded regression models, head pose from geometrical correspondence mapping from 2-D points in the image plane to 3-D points in the head model, horizontal gaze surrogate based on geometrical formulation of the eye ball and iris position, vertical gaze surrogate based on openness of the upper eye lids and appearance descriptor, and finally, a 9-class gaze zone estimation from naturalistic driving data driven random forest algorithm.

We developed a framework to model driver's gaze behavior by representing the scanpath over a time period using glance durations and transition frequencies. As a use case, we explore the driver's scanpath patterns during maneuvers executed in freeway driving, namely, left lane change maneuver, right lane change maneuver and lane keep. It is shown that condensing temporal scanpath into glance durations and glance transition frequencies leads to recurring patterns based on driver activities. Furthermore, modeling these patterns show predictive powers in maneuver detection up to a few seconds a priori and show a promise for developing gaze guidance during take over requests in highly automated vehicles.

We introduce a framework to model the spatio-temporal movements of head, eyes and hands given naturalistic driving data of looking-in at the driver for any events or tasks performed of interest. As a use case, we explore the temporal coordination of the modalities on data of drivers executing maneuvers at stop-controlled intersections; the maneuvers executed are go straight, turn left and turn right. In sequentially increasing time windows, by training classifiers which have the ability to provide discriminative quality of its input variable, the experimental study at intersections shows which type of, when and how long distinguishable preparatory movements occur in the range of a few milliseconds to a few seconds.

We introduce one part of the Vision for Intelligent Vehicles and Applications (VIVA) challenge, namely, the VIVA-face challenge. VIVA is a platform designed to share naturalistic driving data with the community in order to: present issues and challenges in vision from real-world driving conditions, benchmark existing vision approaches using proper metrics and progress the development of future vision algorithms. With a special focus on challenges from looking inside at the driver’s face, we provide information on how the data is acquired and annotated, and how methods are benchmarked, compared and shared on leaderboards.

Finally, we propose de-identification filters for protecting the privacy of drivers while preserving sufficient details to infer driver behavior, such as the gaze direction, in naturalistic driving videos. We implement and compare de-identification filters, which are made up of a combination of preserving eye regions and distorting the background, to show promising results. With such filters, researchers may be more inclined to publicly share deidentified naturalistic driving data. The research community can then tremendously benefit from large amounts of naturalistic driving data and focus on the analysis of human factors in the design and evaluation of intelligent vehicles.