In audiovisual information processing, auditory information may interfere with eye movement planning in visual processing due to competition for attentional resources. Here we hypothesize that this interference may be mitigated in the recognition of emotions involving strong audiovisual coupling. Participants judged the emotion of a talking head video under audiovisual, video-only, and audio-only conditions. While participants generally performed the best in the audiovisual condition, their eye movement pattern did not change significantly across the three conditions except for the recognition of disgust. In disgust recognition, eye movements in the audiovisual condition were less eyes-focused than the video-only condition, and the larger the difference, the less the audiovisual advantage in performance. Disgust recognition develops later in life and may involve weaker audiovisual coupling. Accordingly, our results suggest that whether emotional voice information facilitates emotion recognition without interfering with eye movement planning depends on the strength of audiovisual coupling in emotion processing.

It remains unclear whether multimedia facilitates or impairsknowledge acquisition. Here we examined whether subtitlesand video content facilitate comprehension of documentariesconsisting of statements of facts and whether thecomprehension depends on participants’ cognitive abilitiesand eye movement strategies during video watching. Wefound that subtitles facilitated comprehension regardless ofparticipants’ cognitive abilities or eye movement strategiesfor video watching. In contrast, with video content but notsubtitles, comprehension depended on participants’ auditoryworking memory, task switching ability, and eye movementstrategy. Through the Eye Movement analysis with HiddenMarkov Models (EMHMM) method, we found that acentralized (looking mainly at the screen center) eyemovement strategy predicted better comprehension asopposed to a distributed strategy (with distributed regions ofinterest) after contributions from cognitive abilities werecontrolled. Thus, whether video content facilitatescomprehension of documentaries depends on the viewers’ eyemovement strategy in addition to cognitive abilities.

Recent research has suggested that the visual span in stimulus identification can be enlarged through perceptual learning. Since both English and music reading involve left-to-right sequential symbol processing, music-reading experience may enhance symbol identification through perceptual learning particularly in the right visual field (RVF). In contrast, as Chinese can be read in all directions, and components of Chinese characters do not consistently form a left-right structure, this hypothesized RVF enhancement effect may be limited in Chinese character identification. To test these hypotheses, here we recruited musicians and nonmusicians who read Chinese as their first language (L1) and English as their second language (L2) to identify music notes, English letters, Chinese characters, and novel symbols (Tibetan letters) presented at different eccentricities and visual field locations on the screen while maintaining central fixation. We found that in English letter identification, significantly more musicians achieved above-chance performance in the center-RVF locations than nonmusicians. This effect was not observed in Chinese character or novel symbol identification. We also found that in music note identification, musicians outperformed nonmusicians in accuracy in the center-RVF condition, consistent with the RVF enhancement effect in the visual span observed in English-letter identification. These results suggest that the modulation of music-reading experience on the visual span for stimulus identification depends on the similarities in the perceptual processes involved.

Holistic processing (HP) and hemispheric lateralization are both expertise markers of object recognition. For example, expertise in face and sub-ordinate object perception is shown to be associated with HP and stronger right hemispheric lateralization. However, HP is modulated by experiences of selective attention to parts such as writing experiences of Chinese characters (Tso, Au, & Hsiao, 2014) and drawing experiences of faces (Zhou et al., 2011). Meanwhile, hemispheric lateralization is associated with the decoding strategy employed in object recognition, such as left hemispheric lateralization for reading alphabetic scripts and right hemispheric lateralization for reading logographic scripts. This study aims at training participants to recognize the same sets of artificially-created scripts using either wholeword (Logographic) or grapheme-to-phoneme (Alphabetic) approaches. We found that both approaches induced strong HP, though the alphabetic approach induced stronger left hemisphere advantage than the logographic approach. This training study demonstrates that HP and hemispheric lateralization are separate processes that are associated with different perceptual mechanisms.

Hidden Markov models (HMM) can describe the spatial andtemporal characteristics of eye-tracking recordings incognitive tasks. Here, we introduce a new HMM approach.We developed HMMs based on fixation locations and we alsoused image information as an input feature. We demonstratethe benefits of the newly proposed model in a facerecognition study wherein an HMM was developed for everysubject. Discovery of regions of interest on facial stimuli isimproved as compared with earlier approaches. Moreover,clustering of the newly developed HMMs lead to very distinctgroups. The newly developed approach also allowsreconstructing image information at each fixation.

Here we investigated how music reading experience modu-lates visual spans in language reading. Participants wereasked to identify music notes, English letters, Chinese charac-ters, and novel symbols (Tibetan letters) presented at randomlocations on the screen while maintaining central fixation. Wefound that for music note reading, musicians outperformednon-musicians at some peripheral positions in both visualfields, and for English letter reading, musicians outperformednon-musicians at some peripheral positions in the RVF butnot in the LVF. In contrast, in both Chinese character andnovel symbol reading, musicians and non-musicians did notdiffer in their performance at peripheral positions. Since bothmusic and English reading involve a left-to-right reading di-rection and a RVF/LH advantage, these results suggest thatthe modulation of music reading experience on visual spans inlanguage reading depends on the similarities in the cognitiveprocesses involved.

Holistic processing (HP) has been proposed to be a characteristic of right hemisphere (RH) processing. Here we test this claim using the divided visual field paradigm with Chinese character stimuli. HP is assessed through the composite paradigm, which is commonly used in perceptual expertise research. We found that in novice Chinese readers, a standard HP pattern emerged only in the left visual field/RH but not in the right visual field/left hemisphere, consistent with the analytic/ holistic hemispheric dichotomy in the literature. However, in expert Chinese readers, neither visual field showed the HP pattern, consistent with the finding that reduced HP is an expertise maker for Chinese character recognition. Thus, the RH does not always employ holistic processing; it depends on the perceivers’ experience with the stimuli. This is the first study demonstrating that expertise with a visual object type can modulate hemispheric difference in HP.