Saturday, April 13, 2019

New paper: Your brain on comics

I'm very excited to announce the publication of my newest paper,"Your brain on comics: A cognitive model of visual narrative comprehension" in Topics in Cognitive Science. This journal issue is actually a themed issue edited by me about visual narratives, and this paper is my personal contribution.

This paper in many ways is a culmination of about 10 years of experimental research asking "how do we comprehend a sequence of images?" Much of this work comes from my studies measuring people's brainwaves while they read comics, but it integrates this work with research from fields of discourse, event cognition, and other related disciplines. Here, I tie this work together in a cognitive model to provide an explanation for what happens in the brain when you progress through a sequence of images. My emphasis on brain studies gives the overall endeavor a neurocognitive focus, although the model itself is not specific to the brain.

The primary paper focuses on the evidence for two levels of representation in processing a sequence of images: a semantic structure, that computes the meaning, and a narrative structure, which organizes and presents that meaning in sequencing. In addition, I discuss how these mechanisms are connected to other aspects of cognition, like language and music processing, and I discuss the role of expertise and fluency in comprehending sequential images.

Overall, this is the first full processing theory of visual narrative comprehension, making it a significant marker in the growth of this research field.

The paper is readable online with Open Access, though a downloadable pdf is available here, and via my downloadable papers page. Here's the abstract:

The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative-Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incur costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain-general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences.

Cohn, N. (2019). Your brain on comics: A cognitive model of visual narrative comprehension. Topics in Cognitive Science. doi:10.1111/tops.12421

Friday, April 05, 2019

Knowing the rules of comic page layouts

One of my more engaged-with blog posts of recent memory reviewed the data for whether the panel arrangement on the right was “confusing.” So, here’s a post with some additional thoughts on this and the “rules” of comic page layouts**…

First off, let me remind people that I've given this layout a name: When you have a vertical stack of panels next to a tall panel, I call it "blockage." You can find terms (and science!) related to page layout in my book and my scientific papers (also linked throughout).

Most of the claims I make about page layouts are based on the experiments that I and others have done about them. For this layout, the key experimental findings came from two studies presenting people with empty page layouts, and then asking them to choose the order that they would read the panels.

We found that for blockage layouts, around 90% say “down”. Or, conversely put, less than 10% of choices in these situations followed the "left-to-right-and-down" Z-path that follows the order of written text. As I said in my previous blog post, this rate is essentially the inverse of what we find for pure grids. In simple grids, we find 90% of responses choose to follow the Z-path (i.e., go right) instead of choosing other paths.

Now, one criticism people have about these studies is that they don't have content in the layouts. Yes, these experiments presented empty panels, which might be different than if content is included. But, there's a good reason for this: the question we were asking wasn’t “how do people read these layouts?” but rather “what are people’s preferences for ordering these layouts?” Having no content works just fine for doing good science and factoring out confounding variables, and it answers our question of whether people have preferences for orders: yes they clearly do.

So, these results show that readers have a preference for the proper reading direction. In other words, the “rule” in their minds is that, they should read downward in blockage layouts. You might think that the “rule” of reading comic page layouts is “left-to-right and down”, like text, and thus this layout is confusing. But, that’s not the rule. I’ll explain this more in a bit…

When I say that “this layout is not confusing”, I mean that readers have these clear intuitions for what to do in these situations. The layout itself is not confusing, since people know what to do with it. What gives confusion then, is when creators don’t know or don’t obey this "go downward" rule, and still use layouts where blockage is read to the right. This could feasibly create confusion, since it treats this layout as “neutral” or like there isn’t a rule for its order.

However, there is a clear rule for it, and thinking it’s neutral is wrong according to the experimental results for what people say their preferences are. Grids aren’t used as if right and down are equal choices (though they’re even more physically ambiguous), and nor should this layout.

Certainly a creator can manipulate the reading path by using the content or balloons to go in a different directly. They do this all the time in effective and creative ways even with grids, like in the layout to the left. But, doing it against the downward path in blockage layouts have to be recognized as “breaking the rule” with artistic intent.

So, why isn’t “left to right and down” the real rule of layout? Well, it’s *one* rule in comic page layouts, but it’s just a surface choice within a broader overarching set of rules/principles.

Readers don’t just read a comic page to just go from panel to panel along the “surface” of the canvas making choices like right, down, etc. While it is likely that surface features like balloons and bubbles can "direct" the eye, layouts themselves have rules that are not dependent on these surface features, as demonstrated by the consistent results using empty layouts.

(Note: To my knowledge, there are no controlled experimental results showing that content directs readers' eyes through layouts. There is one non-controlled study that has some hints about this though.)

Here’s the actual rules of layout: Readers go through layouts guided by a desire to create grouped structures out of panels. The surface decisions that they make are on the basis of alignments between the edges of panels, but these choices are subservient to the larger goal of making hierarchic groupings.

I argue that these grouping mechanisms are what underlie readers choices when they move from panel to panel. This may involve some surface level rules, but there is an overarching principle I call “Assemblage” that has four basic sub-principles:

1. Grouped areas > non-grouped areas
2. Smooth paths > broken paths
3. Do not jump over units
4. Do not leave gaps

The reason so many people agree on going down in this layout is because it facilitates chunking the page into grouped structures, while the rightward path doesn’t, and a rightward path violates the Assemblage principles. This is why it’s a “rule.”

So, if you’re a comic creator, knowing what readers are trying to do while they read can help you design layouts, including how to break those rules with intent if you need to do so artistically.

**This originally appeared as a Twitter thread, and has now been expanded for blog format.