Language has been characterized as a “unique” facet of human cognition with complex syntactic features like anaphora and distance dependencies. However, visual narratives, like comics, have been argued to use similar sequencing mechanisms. These narrative structures include “refiner” panels that “zoom in” on the contents of another panel. Similar to linguistic anaphora, refiners co-referentially connect inexplicit information in one unit (refiner/pronoun) to a more informative “antecedent.” Also, refiners can follow their antecedents (anaphoric) or precede them (cataphoric) with either proximal or distant connections. We explored these constraints of order and distance on visual narrative refiners by measuring event-related brain potentials (ERPs) to wordless comic strips. Anaphoric refiners evoked late sustained negativities (Nref) while distant anaphoric refiners attenuated N400s compared to all others, and all distance dependencies evoked leftward negativities. These responses are consistent with (neuro)cognitive responses shown to anaphora in language, suggesting domain-general constraints on the sequencing of referential dependencies.

Sequential images have frequently been used as experimentalstimuli in the cognitive and psychological sciences to exploretopics like theory of mind, temporal cognition, discourse,social intelligence, and event sequencing, among others. Theassumption has been that sequential images provide a fairlyuniversal and transparent stimuli that require little to nolearning to decode, and thus are ideal for non-verbal tasks indevelopmental, clinical, and non-literate populations.However, decades of cross-cultural and developmentalresearch have actually suggested something different: thatsequential image comprehension is contingent on exposureand practice with a graphic system. I here review thisliterature and advocate for more sensitivity to the “fluency”needed to understand sequential images.

Motion lines are the drawn lines trailing behind an object to show movement. Prior work has shown that motion lines aid comprehension by clarifying the direction of motion, but no experimental research has examined how they interact with path segmentation, like a visual sequence in comics. In Experiment 1, participants understood panels with motion lines better than without lines. With motion lines, omitting the path’s source was harder to comprehend than showing the source but having it disappear. Moreover, less experienced comic readers benefited more from showing a whole path with lines in single panels than fluent ones. In Experiment 2, reversed-direction lines were harder to process than normal or omitted lines, but segmentation did not matter. Overall, motion event comprehension can be modulated by path segmentation of lines and comics reading experience.

In discourse, entities that are discontinuous with the current storyline are seen as cues for an event boundary, as they are too irregular to be mapped to the existing scene. However, some instances of discontinuity can maintain coherency, as exemplified in Calvin & Hobbes comics, where visual discontinuities can be resolved by the understanding that they depict Calvin’s imagination rather than actual events. This requires the reconciliation of the original storyworld domain with a private, mental domain (the alternative). In our first experiment, we examined whether switching between domains and/or the nature of the presented domain(s) incurred processing costs. Our second experiment examined whether physical cues such as the contours between the discontinuous entities facilitated processing. The results indicate that switching domains is indeed more costly, despite still being understood as congruous sequences. Moreover, strong similarity in contours aided readers with greater comics proficiency. Overall, our results show that the processing of visual narratives extends beyond mere event understanding and is not universally transparent. Keywords: blending; discontinuity; rhyme; visual narratives

Visual narratives sometimes depict successive images with different characters in the same physical space; corpus analysis has revealed that this occurs more often in Japanese manga than American comics. We used event-related brain potentials to determine whether comprehension of "visual narrative conjunctions" invokes not only incremental mental updating as traditionally assumed, but also, as we propose, "grammatical" combinatoric processing. We thus crossed (non)/conjunction sequences with character (in)/congruity. Conjunctions elicited a larger anterior negativity (300-500 ms) than nonconjunctions, regardless of congruity, implicating "grammatical" processes. Conjunction and incongruity both elicited larger P600s (500-700 ms), indexing updating. Both conjunction effects were modulated by participants' frequency of reading manga while growing up. Greater anterior negativity in frequent manga readers suggests more reliance on combinatoric processing; larger P600 effects in infrequent manga readers suggest more resources devoted to mental updating. As in language comprehension, it seems that processing conjunctions in visual narratives is not just mental updating but also partly grammatical, conditioned by comic readers' experience with specific visual narrative structures.

Like verbal discourse, visual narratives use various techniques to motivate readers to infer unstated events. One technique of sponsoring bridging inferences in visual sequences is to replace climactic events with an “onlooker”, a bystander viewing the off-panel climax reacting with textual and/or bodily cues to varying degrees. In two self-paced reading experiments, we explored how onlookers sponsored inferencing. Experiment 1 compared five types of onlookers ranging from a passive face with no cues to explicit facial or textual cues, and Experiment 2 investigated the effect of zooming in on onlookers’ bodily cues. Less explicit onlookers are processed faster, but at the next panel all onlookers required equal updating costs and zooms elicited further costs. Moreover, textual cues facilitated understanding. Overall, these findings show that complexity and explicitness of information in visual narratives affect subsequent event inferencing. Keywords: backward inferences, explicitness, event cognition

Recognition of animacy is fundamental to human cognition, yet robots complicate this categorization, because they are non-living objects with human-like traits. We examined this categorization of robot animacy using speech balloons from comics, which require connecting to animate “stems” (speakers), or coercing inanimate objects to become animate (e.g., a talking toaster). Participants rated the text-image congruity of silhouettes of humans, inanimate objects, and robots paired with descriptive words placed in either a speech balloon or a label box. Overall, humans and object text-image pairs were rated as more congruent than those with robots. However, a positive correlation suggested human-looking robots with balloons were more congruent than less human-looking ones, but such a graded congruency did not appear with labels. This suggested that speech balloons select for an animate stem compared to labels, but also that intuitions for animacy in robots falls along a gradient depending on their human-like traits.

The neurocognitive processes involved in understanding objects and scenes remains debated, such as the separability of electrophysiological responses thought to index object identification (N300) and semantic access (N400). Yet, studies typically introduce incongruities which evoke N300/N400 patterns, not different deflections. We measured EEG to naturalistic comic strips with panels that “zoomed-in” on scene content. In Experiment 1, zoom and full-scene panels were compared within sequences that were in/congruous to the sequence. Incongruities evoked larger negativities for both the N300 and N400, while zoom panels elicited attenuated N300s yet enhanced N400s. In Experiment 2, zoom and full-scene panels appeared in succession. Both types evoked attenuated N400s when appearing second, benefiting from the repetition effect, but N300s were less negative for zooms than full panels. Across both experiments, these opposite patterns of deflections across components suggest differential processes of object identification (N300) and semantic access (N400) in the processing of visual information.