Tuesday, March 29, 2011

CNS 2011

I will be speaking next week at the Cognitive Neuroscience Society conference in San Francisco. If you're in the area and coming to the conference, I'll be presenting in Slide Session 7, on Tuesday, April 5, 1:00 - 3:00 pm, Grand Ballroom B:

This is your brain on comics: The impact of structure and meaning on sequential image comprehension

Neil Cohn, Martin Paczynski, Phil Holcomb, Ray Jackendoff, Gina Kuperberg; Tufts University

Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing (1, 2) to examine structure and semantics in processing sequential images. Using Cohn’s (3) model of visual narrative, we compared four types of comic strips: 1) Normal sequences with both structure and meaning, 2) Semantic Only sequences (semantic relationships but no structure), 3) Structural Only sequences (structure but no semantic relationships), 4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantics and structure offer advantages to processing. Experiment 2 measured ERPs to the same target panels. The largest N400 appeared in both Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in Normal sequences. This implies that a combination of narrative structure and semantic relationships can facilitate semantic processing (as reflected by the N400). However, the effects of structure alone may be independent of semantics. Taken together, these findings suggest that sequential image comprehension uses a grammar that extends beyond semantic associations between individual frames. The comprehension of graphic narrative is guided by an interaction between structure and meaning, akin to that between syntax and semantics in language.

Wednesday, March 02, 2011

Comic Scramblers

Busy-ness has been in high gear here in visual language research land. I'm working on submitting several papers for publications over the next few weeks, and am over half the way through writing my dissertation proposal. Plus, my wonderful assistants are preparing our next round of experiments...

Speaking of which, one of my assistants, Suzi Grossman, has applied some of my theories to some artistic work. She's made several "scramblers" which generate comics out of random panels. This site has two of them that are made from panels we're using in our research.

The first generator makes completely random strips, so they should feel like a whole lot of gobbledegook. The second generator actually plays with the structure of the narrative though: while certain pieces remain constant, others with particular traits are exchanged for each other. (Press the first green button for new strips. Press the second to swap out that single panel)

She takes this one step further with this project. Here, the generator draws from several different comic strips, but organizes the panels into a coherent narrative, though it might not make sense! I've actually wanted to do a create project like this for some time, so it's great she's actually done it.

Check them out!