I'm very excited to announce that I have a new paper out in the latest issue of Neuropsychologia on how the brain understands sequences of images, "The grammar of visual narratives: Neural evidence for constituent structure in visual narrative comprehension." This was actually one of my dissertation projects, so I'm very happy to finally have it published.
In this paper, I provide evidence that sequential images are processed by creating "constituents"— groupings of panels—rather than just "transitions" between the meanings of panels. Furthermore, disrupting these groupings in the "narrative grammar" of sequential images evokes the same brain responses as when you violate the grammar of sentences.
You can download the full paper here (direct link to pdf). I've also made a video summarizing the paper:
Constituent structure has long been established as a central feature of human language. Analogous to how syntax organizes words in sentences, a narrative grammar organizes sequential images into hierarchic constituents. Here we show that the brain draws upon this constituent structure to comprehend wordless visual narratives. We recorded neural responses as participants viewed sequences of visual images (comics strips) in which blank images either disrupted individual narrative constituents or fell at natural constituent boundaries. A disruption of either the first or the second narrative constituent produced a left-lateralized anterior negativity effect between 500 and 700 ms. Disruption of the second constituent also elicited a posteriorly-distributed positivity (P600) effect. These neural responses are similar to those associated with structural violations in language and music. These findings provide evidence that comprehenders use a narrative structure to comprehend visual sequences and that the brain engages similar neurocognitive mechanisms to build structure across multiple domains.
Cohn, Neil., Jackendoff, Ray., Holcomb, Phillip. J., & Kuperberg, Gina. R. (2014). The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension. Neuropsychologia, 64, 63-70. doi: 10.1016/j.neuropsychologia.2014.09.018