From moment to moment, we perceive objects in the world as continuous despite fluctuations in their image properties due to factors like occlusion, visual noise, and eye movements. The mechanism by which the visual system accomplishes this object continuity remains elusive. Recent results demonstrate that the perception of low-level stimulus features such as orientation and numerosity is systematically biased (i.e. pulled) towards visual input from the recent past. The spatial region over which current orientations are pulled by previous orientations is known as the continuity field, which is temporally tuned for the past 10-15 seconds. This perceptual pull could contribute to the visual stability of low-level features over short time periods, but it does not address how visual stability occurs at the level of complex objects or during visual discontinuities. Here, we examined whether the visual system facilitates stable perception by biasing current perception of objects towards recently seen objects, and whether it operates across disruptions in visibility. First, we used psychophysics to show that the perception of face identity is systematically biased towards identities seen up to several seconds prior, even across changes in viewpoint. This perceptual bias did not depend on subjects’ prior responses or on the method used to measure identity perception. Next, we tested whether this serial dependence, or positive perceptual pull, helps stabilize perceived expression. To test this, observers judged random facial expressions (ranging from happy to sad to angry). We found a pull in perceived expression toward previously seen expressions, but only when the 1-back and current face shared the same identity. Finally, we examined whether the continuity field helps maintain perceived object identity during occlusion. Specifically, we found that the perception of an oriented Gabor that emerged from behind an occluder was significantly pulled towards the random (and unrelated) orientation of the Gabor that was seen entering the occluder. Importantly, this serial dependence was stronger for predictable, continuously moving trajectories, compared to unpredictable ones or static displacements. While this serial dependence in object perception manifests as a misperception, it is adaptive: visual processing echoes the stability of objects in the world to create perceptual continuity. These results provide further evidence for the existence of an object-selective continuity field that helps maintain perceived object stability over time and across visual noise or disruptions.

In a brief glance, we experience a rich percept of our visual environment, despite extremely limited visual capacity. What contributes to this seemingly paradoxical phenomenon? We propose that ensemble perception, the ability to extract statistical information from groups of redundant objects, is a substantial contributing factor to rich, complex visual perception. In support of this proposal, we demonstrate that ensemble perception includes high-level visual percepts, such as multi-viewpoint representations —not merely low-level or texture perception.

We also demonstrate that ensemble perception can bypass attentional and capacity bottlenecks, and successfully function when detailed, single item analysis is unfeasible. Finally, we demonstrate that ensemble representations are functionally beneficial to visual perception, conferring precision unmatched by single-item analysis and effectively operating when other visual processes are impaired. Our investigation of ensemble coding under different visual conditions and within different patient populations provides new evidence that ensemble codes are fundamental components of conscious visual perception

Despite the exciting progress in computer vision in the field of autonomous driving, understanding efficiently which cues or objects are the most crucial ones in a crowded traffic scene is still a big challenge. Human drivers can quickly identify the important visual cues or objects in their blurry periphery vision and then make eye movements to direct their more accurate foveal vision to the important regions. Therefore, driver eye movements may be what computer vision can borrow from human vision to make autonomous driving systems better at locating and understanding the important regions of crowded traffic scenes. Meanwhile, the large-scale datasets and advanced object recognition algorithms that emerged in the field of autonomous driving make it possible to study classical human vision science problems in natural driving situations.

Here, we used driver eye movements to improve autonomous driving models and studied visual crowding—the bottleneck of human object recognition—in realistic driving situations through driver eye movements. First, we developed a new protocol that collects driver eye movements in an offline manner for large-scale driving video datasets. We built a deep neural network that predicts human driver gaze from dash camera videos for various driving scenarios. Our model outperformed the current state-of-the-art model. Furthermore, we incorporated the driver gaze prediction model into an autonomous driving model to make a new periphery-fovea multi-resolution driving model that predicts vehicle speed from dash camera videos. This model combines low-resolution input of the whole video frames and high-resolution input from predicted gaze locations to predict vehicle speed. We show that the added human gaze significantly improves the driving accuracy and that our periphery-fovea multi-resolution model outperforms a uni-resolution periphery-only model that has the same amount of floating-point operations. Finally, we studied visual crowding in driving situations. We show that crowding occurs in natural driving scenes and that the degree of crowding correlates with altered saccade localization in realistic driving-like situations. Together, these studies demonstrate the application of driver eye movements in making safer and more efficient autonomous driving models and show strong evidence of visual crowding in driving situations via the analysis of driver eye movements. These studies also present examples of combining human vision and computer vision to get mutual benefits from both fields.

Emotion recognition is an essential human ability critical for social functioning. It is widely assumed that identifying facial expression is the key to this, and models of emotion recognition have mainly focused on facial and bodily features in unnatural, static, or decontextualized conditions. However, an individual’s face and body are usually perceived within a meaningful context, not in isolation. The visual context, therefore, may provide useful or even necessary information when interpreting emotion. Here, we investigated the role of visual context in dynamic emotion recognition. First, we developed a novel method, "inferential affective tracking (IAT)", to reveal and quantify the contribution of visual context to affect (valence and arousal) perception. We show that when characters’ faces and bodies were masked in silent videos, viewers inferred the affect of the invisible characters successfully and in high agreement, based solely on visual context. We further show that the context is not only sufficient but also necessary to accurately perceive human affect over time, as it provides a substantial and unique contribution beyond the information available from face and body. Next, we tested the efficiency of IAT by measuring the speed of recognizing emotion from contextual information alone. Using cross-correlation analyses, we found that inferring affect based on visual context alone is just as fast as tracking affect with all available information including face and body. We further demonstrated with empirical evidence that this approach has high precision in detecting a sub-second temporal lag. Finally, we extended and adapted the IAT method to test categorical emotion perception rather than affect. This method is very similar to the IAT technique and so we call it “inferential emotion tracking (IET)”. Using IET, we show that the presence of visual context can override interpreted emotion categories from face and body information. Strikingly, we find that visual context determines perceived emotion nearly as much and as often as face and body information does. Taken together, these experiments reveal that emotion recognition is, at its heart, an issue of context as much as it is about faces. Seemingly complex context-based emotion perception is far more efficient than previously assumed.