Monday, July 20, 2015

Comic-Con 2015: The Scientific Study of The Visual Language of Comics

Thanks to my good friend Dan Christensen, I here present my talk from Comic-Con 2015, "The Scientific Study of The Visual Language of Comics." This was my introductory talk for a panel that consisted of three of undergraduates who had been working with me, and provides an overview how I believe that research on comics—or rather, the visual language used in comics—should progress. Alas, note there are a few blips where the video jumps due to some technical difficulties, but only a few seconds are lost.

For those wishing to follow up on things mentioned in the video... Note that my book is available here, and the experiments/diagnostics discussed are available in papers here. In particular, for more advocacy of methods of scholarship about comics, I recommend the paper "Building a better comic theory" (pdf). My experiment about page layouts is in "Navigating Comics II" (pdf). Prior works using corpus analyses are in the "Cross-cultural" papers section.


Tuesday, July 14, 2015

ComicCon 2015 postscript

As always, I had a great time at ComicCon this year! I thought my panel on "The scientific study of the visual language of comics" went great, and I greatly enjoyed talking with everyone who came by the booth to chat with me and check out my book. I am also quite thankful to Reading with Pictures for kindly hosting me at their booth!

In my panel, I gave an overview of how to do "scientific studies" of comics, taking the hard line that this is the only way we can truly get ahold of how the visual language used in comics are structure and comprehended. Much of it echoed the argument from my "Building a better comic theory" paper, along with hints from my introduction to my upcoming book, The Visual Narrative Reader

After this, three of my students each presented their own projects, including an experiment about hierarchy in page layout (Barak Tzori), and coding studies examining how American superhero comics have changed from the 1940s to the present with regards to page layout (Kaitlin Pederson) and text-image interactions (Ryan Taylor).

They totally crushed it! One way I could tell that their presentations were well-received was that most of the questions were directed at them, not me. I was super proud of them, and may try to make this a regular thing at ComicCon.

We attempted to record the presentation, so I'm looking into ways to put that online for all to see parts if not the whole thing. More to come!

Monday, June 15, 2015

The non-universality of cartoony images and comics

There are many who assume that cartoony images and the ability to understand sequential images is universal. The 1978 study "Communicating with Pictures in Nepal" by Fussell and Haaland reports on a study exploring these issues...

This study examined the understanding of images by indiivduals in Nepal. The researchers desired to communicate things related to nutrition, hygeine, reforestation, water supply construction, etc. as part of a UNICEF effort and assumed that wordless pictures would be an effective method. They therefore carried out these studies as a way to confirm that these intuitions were true by presenting 410 Nepalese individuals with drawings and asking for their interpretations. They quickly found that assumptions of universality were wrong.

First they report how Nepalese individuals understood different visual styles by asking them to interpret different types of images (for example, of people, like the image to the right). They tested photos, "blackout" photos with backgrounds omitted, highly detailed line drawings, semi-detailed line drawings with no shading, silhouettes, and cartoony schematic figures. They found that the content of different visual styles were recognized at very different proportions. People "accurately" recognized the content of cartoony (stick figure-esque) styles the worst (49%), while blockout photos without backgrounds (67%) and highly detailed line drawings (79%) were the best.

They had significant deficits understanding many aspects of single images, even images of faces with simple emotions in cartoony styles (happy - 33%, sad - 60%). They had even more difficultly related to actions (only 3% understood an image trying to convey information about drinking boiled water). Some respondents had radically different interpretations of images. For example, a fairly simple cartoony image of a pregnant woman was interpretted as a woman by 75%, but 11% thought it was a man, and others responded that it was a cow, rabbit, bird, or other varied interpretations. They also had difficulty understanding when images cut off parts of individuals by the framing of a panel (such as images of only hands holding different ingredients).

It's worth noting that when looking at the images in the paper, they did not seem overly "poorly drawn" or ambiguous, as the image above shows. By American standards, they were fairly straightforward and drawn in a simple but clear manner. So, it's not just that they were "bad drawings."

Sequences of images fared no better. A two-image comparison of a mother bottle-feeding vs. breast feeding children was only recognized by 19% as being about bottle feeding at all, while only 3% recognized that the image pair was making a comparison.

They also had no assumptions about a left to right (or reversed) reading order, with less than 50% going in this intended order. With a 3-image sequence, some even started in the middle. Even with those who read them in order, most did not understand the connections between images.  The authors note that individuals' degree of literacy was highly predictive of linear reading orders (though it's unknown whether they could also understand those connections).

They state their biggest lesson echoed "Alan Holmes after a study carried out in Kenya in 1961-2: 'It is never safe to act on assumptions as to what people will or will not understand visually without first testing the assumptions.'"

The remainder of the paper discusses efforts to improve instructional papers based on feedback from Nepalese.

In all, these findings are similar to others showing that cartoony images and sequential image understanding are not "universal" without exposure to an external cultural system. From the visual language perspective, these results are expected: one must have exposure and practice to a visual language—just like any other language—in order to understand it.

ResearchBlogging.orgFussell, Diana & Ane Haaland. (1978). Communicating with Pictures in Nepal: Results of Practical Study Used in Visual Education. Educational Broadcasting International, 11 (1), 25-31

Tuesday, May 19, 2015

Review: Unflattening by Nick Sousanis

Nick Sousanis’ recent book Unflattening has been receiving praise for its freeing message and artistic execution. The book was Sousanis’ doctoral dissertation, and in being a graphic work, it thus embodies its message of attempting to break through the confines of the "flatlands" of received viewpoints that unconsciously pervade the ways we see the world. This advocacy specifically targets the hegemony of words, with the freeing power of images—particularly through “comics."

In these regards, the book deserves all the praise it has received. The message is a good one, and the execution is wonderful and sensory. It is a pleasure to read, and nicely balances the artistic and expository, accessible and academic. Clearly, Sousanis knows his craft and carries it out effectively. It also succeeds quite well as just an example of graphic non-fiction, especially without relying on a "narrator character" like McCloud (and included). For these reasons alone the book is worth reading.

For me, the book resonated with my personal experience, but in doing so also betrayed some limitations to its own core message by upholding its own flatlands throughout. I'll spend the rest of this review focusing on these.

The book's advocacy to look beyond the limited viewpoints that one holds both unconsciously and unaware is a great message. In truth, at many times I felt it was speaking directly about my work: In attempting to teach about the theory of visual language, I often feel like Sousanis' referenced example of the sphere telling squares that there is something beyond the flat 2D world as they understand it. As stated in the introduction to my book, I offer an alternative view of the world related to language, drawing, and “comics", and am trying to teach others to see through my eyes.

For example, I cringe whenever I hear someone summarize that I believe “comics are a language,” because it is the exact opposite of what I actually believe. And, it shows that they did not fully understand the alternative viewpoint I offer, but rather pushed my spherical message into the square holes they already held.

It was related to my research that my few criticisms arose though...

First, despite arguing for an interdependence between text and image in their expressive capacities, I found it ironic that only until late in the book does text not lead this dance, and not for long. The images throughout are certainly not negligible, but they mostly enhance, supplement, and enrich the message beyond what text could serve on its own. The text still provides the primary weight of meaning throughout. Rarely are there comprehensible portions of images where the text could be omitted entirely, but the same does not seem to be said for the text (at least, not until the last three chapters). However, I would expect this from a more expository and academic work like this, so it's less of a criticism than an observation.

Second, it was also curious that in its discussion of “comics,” Unflattening most retained what I would say is the “flatland” viewpoint of what they “are” and how they work. To those unfamiliar with comics (and/or their scholarship), this message might be revelatory, but it mostly retains the message reinforced by the “party line” of standard beliefs about “comics” and drawing:

Comics "is" a medium that transcends cultural boundaries, historical periods, etc.

While language is a channel that constrains thought, drawings are freeing, because they encroach on “perception.” Drawings thus get closer to actual pure thought in a unique and individualized way that is not constrained by the memorized patterns of language.

These are the standard perspectives upheld throughout most theories of “comics” and “visual communication,” and Sousanis’ references support these views.

Yet, my work argues that all of these viewpoints are misplaced, and uphold a “flatland” of their own that does not square with our actual cognition or with much of the research on drawing. I would argue that my theory of visual language provides the “upwards not northwards” that Sousanis strives for in this regard...

Drawings are not a siphon for our perception, they reflect entrenched and learned cognitive patterns stored in our memories just as much as language, because they are built just like language. For example, drawing is not about re-presenting perception, but about learning and producing patterned graphic schemas in order to express our concepts. If you don't learn enough patterns, you won't draw proficiently. This makes drawing less like perception, and more like language: both involve stored information in memory. As I argue in my paper “Framing ‘I can’t draw,’” the assumptions about "drawing as tied to perception" actually limit people’s ability to learn to draw, and thereby limit their ability to carry out the mission Sousanis advocates.

Along these lines, Sousanis nicely evokes Lakoff and Johnson’s ideas of conceptual metaphor and Fauconnier and Turner’s ideas of conceptual blending in language. Yet, he does not mention that drawings also use these same conceptualizations (see for example work by Forceville, who will have a summary chapter in my upcoming book).

Furthermore, contrary to the received wisdom, I argue that "comics" "is" not a medium, but rather "comics" are a social construct in which we use two methods of communication: writing (a convergence of spoken language into the visual) and drawing (natural visual language). The union of these things is not “comics” but is our default capacity for multimodal expression, which often happens to appear in a social construct we call “comics”—but not always. As Horrocks’ pointed out, McCloud and others conflated the notion of (sequential) drawing and/or writing into being "comics", a definition constrained by their own flatlands.

Whether you buy into my theories or not, by collapsing these independent but intertwined facets of expression (writing, drawing) into a single construct (“comics”), Sousanis betrays his own mission of attempting to break apart limiting conceptualizations. As a unified thing tied to its stereotypes, it becomes constrained, rather than freed by the unlimited potential of just writing and drawing unbound to such a social construct.

Thus, overall I do recommend the book and wholeheartedly agree with Sousanis’s messages, especially for using text (written/spoken language) and image (visual language) in concert. Indeed I would go further to say that this is our default cognitive human orientation for expression (plus gesture/sign language). However, in its framing of this position, Unflattening retains its own “flatland” that is strived to be overcome, which perhaps in part further speaks to the overall message that escaping one’s frames of knowledge is harder than we might realize.

Thursday, March 12, 2015

New paper on comic page layouts

I'm excited to announce that my paper, "Navigating Comics II" on people's preferences for moving from panel-to-panel in comic page layouts is now published in the latest issue of Applied Cognitive Psychology! This project was undertaken by my student (and co-author), Hannah Campbell, for my course on the Cognition of Comics at UC San Diego.

This project is a follow up to my previous study looking at participants' preferences for how to navigate through comic page layouts, also discussed in my book on the visual language of comics. While we tested several different features of page layouts, here's a graphic version of our most interesting finding:

You can find the paper at my Downloadable Papers page, or directly here (pdf).


Although readers typically believe that comic page layouts should be read following the left to right and down ‘Z-path’ inherited from written language, several spatial arrangements can push readers to deviate from this order. These manipulations include separating panels from each other, overlapping one panel onto another, and using a long vertical panel to the right of a vertical column to ‘block’ a horizontal row. We asked participants to order empty panels in comic page layouts that manipulated these factors. All manipulations caused participants to deviate from the conventional Z-path, and this departure was modulated by incremental changes to spatial arrangements: The more layouts deviated from a grid, the less likely participants were to use the Z-path. Overall, these results reinforce that various constraints push comic readers to engage with panels in predictable ways, even when deviating from the traditional Z-path of written language.


Cohn, Neil and Hannah Campbell. 2015. Navigating comics II: Constraints on the reading order of page layouts. Applied Cognitive Psychology. 29: 193-199

Thursday, February 05, 2015

New paper: Notion of the Motion

I'm excited to say that my paper, "The notion of the motion: The neurocognition of motion lines in visual narratives" with Steve Maher is now published in the latest issue of Brain Research. It examines the comprehension of motion lines in comics. We show that having no lines is worse than having motion lines, but having backwards, anomalous lines is even worse than no lines.  In their context in a sequence of images, processing of these anomalies may activate brain areas typically related to language processing. In addition, this understanding is also modulated by people's experience reading comics, suggesting that they are conventionalized pieces of "vocabulary" in the visual language of comics.

The paper is available here (pdf) and in the "visual vocabulary" section of my Downloadable Papers page along with all other papers about visual language research. A short, graphic summary is readable here.

Full abstract:

Motion lines appear ubiquitously in graphic representation to depict the path of a moving object, most popularly in comics. Some researchers have argued that these graphic signs directly tie to the “streaks” appearing in the visual system when a viewer tracks an object (Burr, 2000), despite the fact that previous studies have been limited to offline measurements. Here, we directly examine the cognition of motion lines by comparing images in comic strips that depicted normal motion lines with those that either had no lines or anomalous, reversed lines. In Experiment 1, shorter viewing times appeared to images with normal lines than those with no lines, which were shorter than those with anomalous lines. In Experiment 2, measurements of event-related potentials (ERPs) showed that, compared to normal lines, panels with no lines elicited a posterior positivity that was distinct from the frontal positivity evoked by anomalous lines. These results suggested that motion lines aid in the comprehension of depicted events. LORETA source localization implicated greater activation of visual and language areas when understanding was made more difficult by anomalous lines. Furthermore, in both experiments, participants׳ experience reading comics modulated these effects, suggesting motion lines are not tied to aspects of the visual system, but rather are conventionalized parts of the “vocabulary” of the visual language of comics.

Cohn, Neil and Stephen Maher. 2015. The notion of the motion: The neurocognition of motion lines in visual narratives. Brain Research. 1601: 73-84

Monday, December 29, 2014

Igor Kordey's "Design of comics"

This morning I found this very fun post by Designboom about a talk given by the Croatian comic art Igor Kordey about his page design and storytelling. The post and Kordey's talk are both interesting and entertaining, and Kordey's pages are simply beautiful. That alone is worth the price of admission. However, what I found really interesting was that he identifies several specific designs and techniques throughout his practice, many of which I've discussed in my book and papers. I could talk about every page he shows in a lot of detail, but I'll break down a few of his points.

First off, many of his layouts simply involve symmetrical inversions of structure. So, if there are three horizontal panels on top, he'll have three vertical ones on the bottom. If there is a blockage arrangement on top (two vertically stacked next to a large panel to their right), then he'll do a reverse blockage layout on the bottom (large panel on left, two vertical stacked to their right).

In other pages he prefers to keep the layout constant, with different types of grids, but then to manipulate the content inside. A recurring theme is to keep the panel content from the same perspective (say, a lateral view) with several non-moving objects and one that moves, while then flipping the perspective in only a few panels of the page. (Notably, doing this in his third example page, it violates the 180ยบ rule between nearly every panel on the page, yet he makes no mention or care of it). Across all of these, we might say he follows a rule of "same-except": they're the same, except different in some particular dimension.

He also describes more specific techniques. In one sequence, he describes how the panels shift back and forth between showing a hunter killing a bird and a politician being assassinated (his example two page spread comes after the pages below—no image available, but it's in the video of his presentation). In film theory, this type of flipping back and forth is called "parallel-cutting" or "cross-cutting," but I've called it "multi-tracking," "polytaxis," or "alternation" (depending on certain features of the alternating panels).

I actually spend some space in my book discussing these types of patterns, and show that they recur in several different types of comics (I have a whole folder full of them from different books). As I mention in the book, they are a good example of an abstract memorized pattern found across sequences of images, rather than just patterns found in the the "vocabulary" of images' content (like how hands or upfixes are drawn).

Another particularly striking page that he discusses uses what he calls a "fishbone" layout, which is depicted in the image below on the right side:

This sequence uses alternating horizontal and vertical panels that descend as one progresses through the page. He says that he's never seen it in any other comic, but I got very excited seeing this page, since I remembered drawing pages like this (and his next page) when I was experimenting with my own comics in college a decade ago. In addition, I discussed a similar page specifically in my paper on the structure of page layouts from the comic Scott Pilgrim:

The layout in the Scott Pilgrim page is structurally the same as Kordey's—also with 9 panels!—but Kordey's uses panels of almost equal sizes, whereas those in Scott Pilgrim shrink with every new panel. The effect in Kordey's makes it a fairly "clean design" and his content mirrors this with the horizontal panels showing images of one character looking up at another, while the vertical panels show their vertical relationship (one on top of the other). In Scott Pilgrim, the shrinking panels also mirror the content—the character is getting off of a bus which then drives off in the distance, so the size of panels reduces as the distance grows greater.

To the left of the Scott Pilgrim page, you can see the actual structure of these pages. Layouts in general are structured so that vertical and horizontal structures embed inside each other (see my book or paper on layout for details). While Kordey emphasizes the back-and-forth between the content in his vertical and horizontal panels, what makes these pages interesting is not just that panels alternate from horizontal to vertical, but that each panel creates a new tier in which the later panels are embedded. Layouts like these create a "right-branching tree structure"—and the descent in the tree diagram shows the feeling of the descent in the layout.

There is no image for it on the page, but his next example in the video is the subsequent page where he reverses the "fishbone" so that the panels expand. This then flips the layout, with a more "expanding" feeling resulting. The structure for this would be the exact reverse of the "right-branching tree" and would instead be a "left-branching tree," which again could directly show why the feeling of "expansion" occurs in its structure.

One of his final pages actually belies the system of embedding panels in layouts, since the borders of the panels overlap each other, similar to the Steranko page to the right (again, discussed in my paper on layout). In fact, his whole page design is the same as the five panels in the upper left of Steranko's layout which look almost like a pinwheel, or as Kordey calls it, the "swastika" layout.

As Kordey suggests, these layouts are much harder to navigate, since they belie the normal rules of reading layouts.** However, the content allows this, since there is almost no progression of time between the panels, meaning that the linear order is less important. In my narrative structure, I would say that these panels are "conjoined" within the same clause and thus can be rearranged in their order with little effect on the coherence of the sequence.

This seems to be Kordey's main point throughout: that the structure of the "storytelling" and the structure of the "layout" should reinforce each other. I think my theory would agree with this, since both narrative and layout create "tree structures," and it makes sense that when both have similar structures, it should create either "easier" processing or maybe a seemingly more aesthetic response (we'd have to do experiments to find out for sure).

And... I'd add that having a well-developed theory of these structures allows us to directly show these insights of a brilliant comic artist like Kordey.

** Note: A supplement to my original page layout paper (pdf) shows that, though the whole the Steranko page poses challenges, comic readers still have consistent patterns by which they preferentially navigate it.

Monday, December 15, 2014

Fall projects and complex theory

Alas, these last few months have flown by with little time to post. After recuperating from my flurry of summer/fall travel, I've been diligently working on a few new projects, especially a new brainwave study. I've also been amassing chapters from various authors for my upcoming edited book on visual narrative, on its way this time next year (more info soon...), and have been submitting lots of new journal articles. Busy times!

Several exciting studies have also been underway with my student researchers, who have coded through various comics of the world (previous works of this type found here). Our projects have so far been looking at page layouts, panel framing, and multimodal interactions. Researchers progress through comics one panel at a time and record properties of the panels, and then we tabulate those numbers and compare samples of comics from different countries, genres, time periods, etc. This is the most effective method of seeing what's going on in the structures of actual comics across the world without hand-waving speculations.

Our focus now is turning to narrative patterns. By February, we're hoping to have coding for roughly 200 comics of multiple genres from at least 5 different countries. This should hopefully begin to tell us the scope of various narrative patterns across cultures and what sort of variation might occur between cultures. This study is by far the biggest scope of its kind, but I consider it only a starting point for more extensive works to come.

The other exciting thing this quarter has been my bi-weekly "advanced visual language seminar" with my student researchers here at UC San Diego. Here, we've been exploring complex aspects of my theory of narrative grammar that have extended beyond what is in my prior publications, especially with regard to the "semantic structures"—the components that describe a sequences' meaning.

Once we established the basic theories, we have put them to use by analyzing various pages from comics, like the Calvin and Hobbes page above. Our recent method has been to randomly choose pages from my database of comics and then analyze it across multiple structures: graphic structure (the lines on the page), page layout, narrative structure, spatial structure, conceptual structure. We then show how all these structures interact.

This can all get pretty complex, like this two page spread from Lone Wolf and Cub, above. Compared to theories like panel transitions, sussing out all this structure is fairly difficult and complicated. However, the structure of sequential images (and the brain!) are indeed complex! And, when considering what each of these components do, they all seem necessary. This is truly an amazing system of communication!  (...and brain!)

Either because or despite this complexity, this practice is actually fairly fun. I think working through these structures may be the best way to appreciate and get a handle on what is actually going on in the structure of this visual language.

For those interested in more, further examples and descriptions of these interactions can be found in my recent review paper, online here. Hopefully papers (and maybe a book?) can better address all this in years to come.

Tuesday, October 14, 2014

Using the theory of narrative grammar

As people have now started reading my book and papers, they've naturally started to try to apply my theories of "narrative grammar" to sequential images found in comics. My "narrative grammar" is a model of how the "storytelling" of sequential images is understood, which extends beyond previous approaches like Scott McCloud's theory of panel-to-panel transitions.

As people have now opened up comics, and tried to use the theory to describe randomly found pages and sequences, they have no doubt discovered that it is not easy. In fact, they may have thrown up their hands in frustration. If this theory is psychologically real then, why is it so hard to analyze sequences? Does this mean the theory is wrong?

No. There are many reasons why analysis may be challenging...

1. Multimodality

First, the theory is designed to account for sequences without text. Once text is introduced, the sequence must balance both structure (i.e., grammar) and meaning in multiple modalities, and it becomes manifestly more complex. I'm hoping to have a paper detailing this out soon.

2. Procedures

Ok, so what about wordless sequences? Just as it would be really difficult to just read a paper about linguistic syntax and analyze sentences, this theory requires some training to do it properly. At the very least, it helps to follow procedures for how to go about analysis.

Even I don't just look at a sequence and immediately know what the analysis is. I go through a series of procedures that tests the structure at each step of the way (these procedures are found in both my book, in Chapter 6, and the section on "diagnostics" in the "Visual Narrative Structure" paper, though not enumerated for how to go through them).

Here's how I train students in my classes and workshops to analyze sequences: the first thing we do is find the Peak panels. The rest of the sequence hangs around the Peaks, so it's the first thing we find. How do we know what is a Peak then? We test panels by trying to delete them (if the sequence is weird without them, then it's likely a Peak) or replacing them with an action star (if it does replace, it's likely a Peak), or deleting everything else except them (Peaks should be able to paraphrase a sequence on their own). From here, other procedures are then used to determine the other categories and the hierarchy of the sequence.

The point being: you can't just look at a sequence and intuit the structure (even me). That's why I describe tests and diagnostics, so that you can do it without just relying on intuition at every step of the way. Procedure matters.

3. Theory as framework

Third, the theory is a framework, not a catch-all. Theories of syntax in language are not "fully formed" when they are written about, and no theory of syntax in any book or any paper—of any linguistic model—is designed to immediately encompass every sequence one could encounter "out of the box." Rather, the theory provides a framework by which to account for the various diversity found in sentences. One then uses the framework (or changes the framework) to describe the various phenomena that are found in actual language use.

I consider a theory to be "good" if it can do two things: 1) account for more phenomena that is found in a structure (here, visual sequences) than other theories, and 2) can be revealed in experimentation to have psychological validity.

Much of syntactic theory about language is not simply finding things in sentences and then describing them using a particular theoretical model. Rather, the examples found in sentence structures are both described with theories of syntax, but also are used to illustrate how they pose challenges to theories of syntax such that those theories must grow and change. Throughout the 1960s and 1970s, much of the "wars" that were fought throughout linguistics had this characteristic—finding various patterns of syntax that would force changes towards preferring one theory of grammar or another.

That said, my theory of "Visual Narrative Grammar" is not meant as an "out of the box" analysis tool that should apply to every sequence of images from a comic based only on the chapter from my book. There are many, many more traits of the theory that have yet to be published, all of which deal with more complicated sequences and the non-trivial issue of combining sequential images with text. What is in my book and papers so far is less than what I even teach in an introductory class on visual language: I have a draft manuscript of over 300 pages (and growing) detailing various phenomena with the theory, most of which hasn't been published yet.

In addition, various sequences should challenge the theory, which is exactly the method I've used for the past 15 years to build the theory in the first place. I've had a theory, then found sequences that force changes to the architecture, and then altered the theory to be able to account for those issues. It's an organic process. The theory gives us a way to discuss and analyze such complexity and see how it might work. It's a framework, not a catch-all, just like all linguistic theories.

This is exactly the opposite of something like panel-to-panel transitions which are based solely on the low-level changes in meaning that occur between images. Such a theory is simple—there will always be meaningful changes between panels, and so it always seems to work. That's its appeal. The problem is that such an approach doesn't explain much of the data, which is far more complex than such a simple approach can manage. Indeed, my approach first started by expanding McCloud's panel transitions, and then altering it as I found sequences that it couldn't handle.

The fact is, the way we express meaning—be it through verbal language, visual language, or their combination—is very complex. There are no simple answers, and we should distrust the simple answers that might be offered. Recognizing this complexity, and building a framework that can let us study it, is the first step to exploring how it is understood.

Thursday, October 02, 2014

New paper: The grammar of visual narratives

I'm very excited to announce that I have a new paper out in the latest issue of Neuropsychologia on how the brain understands sequences of images, "The grammar of visual narratives: Neural evidence for constituent structure in visual narrative comprehension." This was actually one of my dissertation projects, so I'm very happy to finally have it published.

In this paper, I provide evidence that sequential images are processed by creating "constituents"— groupings of panels—rather than just "transitions" between the meanings of panels. Furthermore, disrupting these groupings in the "narrative grammar" of sequential images evokes the same brain responses as when you violate the grammar of sentences.

You can download the full paper here (direct link to pdf). I've also made a video summarizing the paper:

Official abstract:

Constituent structure has long been established as a central feature of human language. Analogous to how syntax organizes words in sentences, a narrative grammar organizes sequential images into hierarchic constituents. Here we show that the brain draws upon this constituent structure to comprehend wordless visual narratives. We recorded neural responses as participants viewed sequences of visual images (comics strips) in which blank images either disrupted individual narrative constituents or fell at natural constituent boundaries. A disruption of either the first or the second narrative constituent produced a left-lateralized anterior negativity effect between 500 and 700 ms. Disruption of the second constituent also elicited a posteriorly-distributed positivity (P600) effect. These neural responses are similar to those associated with structural violations in language and music. These findings provide evidence that comprehenders use a narrative structure to comprehend visual sequences and that the brain engages similar neurocognitive mechanisms to build structure across multiple domains.

Cohn, Neil., Jackendoff, Ray., Holcomb, Phillip. J., & Kuperberg, Gina. R. (2014). The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension. Neuropsychologia, 64, 63-70. doi: 10.1016/j.neuropsychologia.2014.09.018

Thursday, September 11, 2014

New Paper: You're a good structure, Charlie Brown

Wow, big week for new papers! I've got another one officially out now in the latest issue of Cognitive Science, "You're a Good Structure, Charlie Brown: The Distribution of Narrative Categories in Comic Strips" (pdf). This one actually reports on the first studies I did on comics during grad school, way back in 2007! (Sometimes science takes awhile to get out...).

This study examines whether the narrative categories I've proposed for sequential images use patterned roles in sequences. There are four experiments we used:

Experiment 1: People took 4 unordered panels and ordered them into a sequence
Experiment 2: We measured how long people viewed panels in sequences where two panels were reversed in order
Experiment 3: People took 4 unordered panels, ordered 3 into a sequence and deleted 1
Experiment 4: People viewed 3 panel sequences where one panel had been deleted, and guessed where it was deleted from

Across all tasks, we found complementary results for how different types of panels behaved, showing that there are certain "distributional trends" for the roles that panels play in a sequence. For example, panels that were freely chosen to be deleted were not recognized when they were missing, but panels that were not often deleted were noticed when gone.

Plus, some panels can play multiple roles in a sequence, but not all panels have this ability. This confirms that some types of panels are flexible in the role they play in a sequence, but not all panels can go in any location in a sequence. This goes against the idea that "any panel can go in any position" in a sequence and still be meaningful.

Here's the full abstract:

Cohn's (2013) theory of “Visual Narrative Grammar” argues that sequential images take on categorical roles in a narrative structure, which organizes them into hierarchic constituents analogous to the organization of syntactic categories in sentences. This theory proposes that narrative categories, like syntactic categories, can be identified through diagnostic tests that reveal tendencies for their distribution throughout a sequence. This paper describes four experiments testing these diagnostics to provide support for the validity of these narrative categories. In Experiment 1, participants reconstructed unordered panels of a comic strip into an order that makes sense. Experiment 2 measured viewing times to panels in sequences where the order of panels was reversed. In Experiment 3, participants again reconstructed strips but also deleted a panel from the sequence. Finally, in Experiment 4 participants identified where a panel had been deleted from a comic strip and rated that strip's coherence. Overall, categories had consistent distributional tendencies within experiments and complementary tendencies across experiments. These results point toward an interaction between categorical roles and a global narrative structure.

Full Reference:

Cohn, Neil. (2014). You’re a good structure, Charlie Brown: The distribution of narrative categories in comic strips. Cognitive Science, 38(7), 1317-1359. doi: 10.1111/cogs.12116

Monday, September 08, 2014

New paper: Building a better "comic theory"

I'm happy to say that I have a new paper (pdf), "Building a better "comic theory," in the latest issue of the journal Studies in Comics. In this one I critique the existing theories about "how comics are understood" and provide a framework for better research to be undertaken. Longtime readers of this blog will certainly recognize some of my advice for researchers, now presented here in a coherent fashion.

Here's the abstract in full:

Research on the understanding of ‘how comics work’ has grown tremendously over the past twenty years, with more articles and books emerging each year. Much of this research has discussed comparisons between comics and language, and/or has speculated on comics’ cognition. However, much of this research faces limitations, which hamper the seriousness of the endeavour and reflect the youth of this emerging field. This article points out these deficiencies that pervade theories about comics. These include inadequate background research, overly general and unsupportable claims, a lack of adequate evidence, and limitations for research methodologies. To address these concerns, I draw from over 50 years of research from linguistics and cognitive science to inform how the field of ‘comic theory’ can move forward. In particular, I outline two primary ways of progressing with this line of research: (1) explicit manipulation of the component parts of the structure used in comics and (2) cataloguing actual comics for various theoretically relevant phenomena. This data-driven approach is offered as a guiding vision for future works on the understanding of ‘how comics work’.

Full Reference:

Cohn, Neil. 2014. Building a better “comic theory”: Shortcomings of theoretical research on comics how to overcome them. Studies in Comics. 5(1), 57-75