In audiovisual information processing, auditory information may interfere with eye movement planning in visual processing due to competition for attentional resources. Here we hypothesize that this interference may be mitigated in the recognition of emotions involving strong audiovisual coupling. Participants judged the emotion of a talking head video under audiovisual, video-only, and audio-only conditions. While participants generally performed the best in the audiovisual condition, their eye movement pattern did not change significantly across the three conditions except for the recognition of disgust. In disgust recognition, eye movements in the audiovisual condition were less eyes-focused than the video-only condition, and the larger the difference, the less the audiovisual advantage in performance. Disgust recognition develops later in life and may involve weaker audiovisual coupling. Accordingly, our results suggest that whether emotional voice information facilitates emotion recognition without interfering with eye movement planning depends on the strength of audiovisual coupling in emotion processing.

It remains unclear whether multimedia facilitates or impairsknowledge acquisition. Here we examined whether subtitlesand video content facilitate comprehension of documentariesconsisting of statements of facts and whether thecomprehension depends on participants’ cognitive abilitiesand eye movement strategies during video watching. Wefound that subtitles facilitated comprehension regardless ofparticipants’ cognitive abilities or eye movement strategiesfor video watching. In contrast, with video content but notsubtitles, comprehension depended on participants’ auditoryworking memory, task switching ability, and eye movementstrategy. Through the Eye Movement analysis with HiddenMarkov Models (EMHMM) method, we found that acentralized (looking mainly at the screen center) eyemovement strategy predicted better comprehension asopposed to a distributed strategy (with distributed regions ofinterest) after contributions from cognitive abilities werecontrolled. Thus, whether video content facilitatescomprehension of documentaries depends on the viewers’ eyemovement strategy in addition to cognitive abilities.

Here we examined whether one’s perceptual style in viewing own- and other-race faces is associated with performance and bias in social categorization by race, and whether mask use modulates the perceptual style and social categorization effects. We found that Asian participants who adopted more eyes-focused eye movement patterns when viewing Asian faces had a larger bias to judge 50% Asian-Caucasian face morphs as Asian. However, although mask use made participants’ viewing pattern more eyes-focused, it did not change this bias in judging morphed faces, or other-race advantage in social categorization speed. These results suggest that information from the eye region may be sufficient to induce these social categorization effects, and that transient perceptual input change due to mask use does not modulate these social categorization effects. Thus, effects and biases in social categorization may be impervious to mask use. These findings have important implications for social interaction during the pandemic.

Here we used deep neural network + hidden Markov model (DNN+HMM) to provide a computational account for the relationship among holistic processing (HP), face scanning pattern and face recognition performance. The model accounted for the positive associations between HP and eyes-focused face scanning pattern/face recognition performance observed in the literature regardless of the version of the composite task used to measure HP. Interestingly, we observed a quadratic relationship between HP and face scanning pattern, where models being highly eyes-focused or highly nose-focused had lower HP. By inspecting fixation locations and associated attention window size in the model and XAI methods, we found that the eyes- and nose-focused models both developed local and holistic internal representations during training, and their difference was in the temporal dynamics of how these representations were used. Our findings demonstrated how computational modeling could unravel the mechanisms underlying cognition not readily observable in human data.

While saliency-based explainable AI (XAI) methods have been well developed for image classification models, they fall short in comparison with human explanations. Here we examined human explanation strategies for image classification and their relationship with explanation quality to inform better XAI designs. We found that individuals differed in attention strategies during explanation: Participants adopting more explorative strategies used more visual information in their explanations, whereas those adopting more focused strategies included more conceptual information. In addition, visual explanations were rated higher for effectiveness in teaching learners without prior category knowledge, whereas conceptual explanations were more diagnostic for observers with prior knowledge to infer the class label. Thus, individuals differ in the use of visual and conceptual information to explain image classification, which facilitate different aspects of explanation quality and suit learners with different experiences. These findings have important implications for adaptive use of visual and conceptual information in XAI development.

Through comparing autistic and non-autistic adults in learning and recognizing masked faces, we found that although autistic participants generally had poorer face recognition performance than matched controls, the two groups were similarly impaired by mask use. Nevertheless, when viewing masked faces during learning, they showed reduced tendency to look at the eyes and reduced change in eye movement consistency as compared with controls; this was not observed during recognition. Across participants, selective attention ability and flexibility to change face scanning behavior according to mask conditions were two important factors accounting for individual differences in performance. Interestingly, autistic spectrum quotient accounted for additional variance when recognizing masked faces learned also with a mask, suggesting additional influence from one's autistic traits that could have impacted face learning experience during development. Our findings have important implications for identifying vulnerable populations whose face recognition ability may be particularly affected by mask use.

We examined whether mask use had differential impacts on face and facial expression recognition across cultures, as cultures associated with more eyes-focused face scanning strategies may be less affected. Asian and White participants performed face and facial expression recognition with unmasked and masked Asian and White faces. White participants attended more to the eye region in both tasks; however, their performance was less impaired by mask use only in facial expression recognition. In both tasks, individuals adopting more eyes-focused strategies for unmasked faces were less impaired by mask use. Also, participants had larger performance impairment for judging expressions of Asian than White faces, consistent with the finding that they adopted more nose-focused strategies for Asian than White faces. Thus, although individuals from different cultures or expression recognition of different races may be affected differentially by mask use, these effects may be better explained by individual differences in preferred attention strategies.

Here we examined whether eye movement measures in global-local perceptual processing tasks, where abnormalities were typically found in individuals with schizophrenia, could be used to predict schizotypy through Eye Movement analysis with Hidden Markov Models (EMHMM). Using both multiple regression analysis and Gaussian process classifier to predict schizotypy, we found that in addition to longer response times in contour integration, a less consistent eye fixation to locate a stimulus and a more consistent subsequent fixation to start engaging local processing in the embedded figures task predicted high schizotypy. These effects may be related to reduced top-down attention control due to deficient global processing and enhanced local processing bias respectively. In addition, performance in embedded figures could further enhance classification accuracy when being used in conjunction with the above predictors, suggesting the multifactorial nature of the identification problem. These predictors may be important endophenotype markers for schizotypal personality.