This is your brain on comics: The impact of structure and meaning on sequential image comprehension
Neil Cohn, Martin Paczynski, Phil Holcomb, Ray Jackendoff, Gina Kuperberg; Tufts University
Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing (1, 2) to examine structure and semantics in processing sequential images. Using Cohn’s (3) model of visual narrative, we compared four types of comic strips: 1) Normal sequences with both structure and meaning, 2) Semantic Only sequences (semantic relationships but no structure), 3) Structural Only sequences (structure but no semantic relationships), 4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantics and structure offer advantages to processing. Experiment 2 measured ERPs to the same target panels. The largest N400 appeared in both Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in Normal sequences. This implies that a combination of narrative structure and semantic relationships can facilitate semantic processing (as reflected by the N400). However, the effects of structure alone may be independent of semantics. Taken together, these findings suggest that sequential image comprehension uses a grammar that extends beyond semantic associations between individual frames. The comprehension of graphic narrative is guided by an interaction between structure and meaning, akin to that between syntax and semantics in language.
Tuesday, March 29, 2011
I will be speaking next week at the Cognitive Neuroscience Society conference in San Francisco. If you're in the area and coming to the conference, I'll be presenting in Slide Session 7, on Tuesday, April 5, 1:00 - 3:00 pm, Grand Ballroom B: