Monday, September 28, 2015

New paper: Getting a cue before getting a clue

It seems the last few months on this blog have been all about inference generation... I'm happy to say this post is also the case! I'm excited to announce that I have a new paper out in the journal Neuropsychologia entitled "Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension."

This paper examines the brain response to the generation of inference in a particular narrative construction in comics. As far as I know, it's the first neuroscience paper to examine inference specifically in visual narratives. Specifically, our analysis focused on comparing sequences like these:

The top sequence (a) is from an actual Peanuts strip. What is key here is that you never see the main event of the sequence: Linus retrieving the ball. In my narrative structure, this "climactic" state would be called a "Peak." Rather, the image of Charlie watching ambiguously hides this event, but that panel is more characteristic of a "Prolongation" that extends the narrative further without much action.

Contrast this with (b), which has a structure that also appears in several Peanuts strips. Here, the third panel also does not show the main event (the same event as "a") but here the exclamation mark implies at least that some event is happening at least. In my narrative structure, this cue is enough to tell you that this panel is the climax, despite not showing you what the climax is.

We were curious then if the brain distinguishes between these types of sequences which both should require inference (indeed, the same inference) but differ in their narrative structure (spoiler: it does!). You can read a full pdf of the paper here. Here's the full abstract and reference:


Inference has long been emphasized in the comprehension of verbal and visual narratives. Here, we measured event-related brain potentials to visual sequences designed to elicit inferential processing. In Impoverished sequences, an expressionless “onlooker” watches an undepicted event (e.g., person throws a ball for a dog, then watches the dog chase it) just prior to a surprising finale (e.g., someone else returns the ball), which should lead to an inference (i.e., the different person retrieved the ball). Implied sequences alter this narrative structure by adding visual cues to the critical panel such as a surprised facial expression to the onlooker implying they saw an unexpected, albeit undepicted, event. In contrast, Expected sequences show a predictable, but then confounded, event (i.e., dog retrieves ball, then different person returns it), and Explicit sequences depict the unexpected event (i.e., different person retrieves then returns ball). At the critical penultimate panel, sequences representing depicted events (Explicit, Expected) elicited a larger posterior positivity (P600) than the relatively passive events of an onlooker (Impoverished, Implied), though Implied sequences were slightly more positive than Impoverished sequences. At the subsequent and final panel, a posterior positivity (P600) was greater to images in Impoverished sequences than those in Explicit and Implied sequences, which did not differ. In addition, both sequence types requiring inference (Implied, Impoverished) elicited a larger frontal negativity than those explicitly depicting events (Expected, Explicit). These results show that neural processing differs for visual narratives omitting events versus those depicting events, and that the presence of subtle visual cues can modulate such effects presumably by altering narrative structure.

Cohn, Neil, and Marta Kutas. 2015. Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension. Neuropsychologia 77:267-278. doi: 10.1016/j.neuropsychologia.2015.08.026.

Tuesday, September 01, 2015

Inference generating comic panels

Since my last post discussed my new paper on action stars, I thought it would be worth doing a refresher on these types of panels in the visual language of comics. "Action stars" are a type of panel that replaces a primary action of a sequence with a star-shaped flash, which on its own usually represents an impact. In the case of action stars, this representation is blown up so large that it encompasses the whole panel, as in the third panel here:

Interestingly, the "star shaped flash" of action stars does not necessarily just convey an impact—my study has shown that seems to generalize to lots of events even without an impact. One reason might be because the "star shaped flash" representation is also the way to typically represent the "carrier" of sound effects. Sound effects, like "Pow!" do typically—but not always—accompany action stars. So, this representation is technically polysemous between impacts and loud sounds—the same physical representation can have multiple meanings—and in the case of action stars it is a little ambiguous.

The key thing I want to focus on here though is that action stars replace the primary actions of the sequence, and thus cause those events to be inferred. In the example above, you infer that Snoopy is run over by the boys playing football, though you don't see it. This doesn't happen "in between the images," but happens at the action star itself, though you don't know what that event is until the next panel.

I discuss these types of "replacing panels" ("suppletive" in linguistic parlance) quite a bit in my book, The Visual Language of Comics, where I pointed out that not all images can work in this way. For example, the "fight cloud" in (b) does work effectively to replace panels—here meaning specifically a fight, not a general action like action stars. But, not all panels can play this "replacing" role. For example, using a heart to replace a whole panel doesn't work as well (c), even when it's used in a context where it could be possible (d):

So, not all elements can replace actions in panels. Recently, I stumbled on another one though in the comic Rickety Stitch and the Gelatinous Goo where an imp uses magic to transform into looking like a gnome:

Again, a full panel here does not depict the action, but replaces the event, leaving it to be inferred. In this case, the "poof cloud" provides a particularly useful covering for avoiding the representation of the physical transformation (which might be a pain to draw). Instead, this transformation is left to the audience's imagination.

In many cases, the onomatopoeia is not needed for these replacement panels, and I've found examples both with and without text. Similar replacement seems to occur without the visual language of the images (clouds, stars), and with the onomatopoeia alone, as in this Calvin and Hobbes strip:

So, onomatopoeia and these replacing panels can go together, or separately. In all though we seem to have an overarching technique for "replacing actions" with visual and/or verbal information which causes inference for the missing information. In the case of the visual information, it seems we have at least three systematic usages: action stars, fight clouds, and poof clouds. Perhaps there are more?