Thursday, February 28, 2013

Science and Eddie Campbell's rules of comics comprehension

In Eddie Campbell's recent article at The Comics Journal, he described several potential "rhetorical rules" that authors of comics can follow in order to make them more understandable to inexperienced readers. In a previous post, I tackled the broader issue of what factors go into limitations to understanding. In this post, I discuss what scientific research tells us about his actual stated rules.

Rule 1: All the information necessary to understand the drama of a sequence must be contained in every panel of the sequence.

This rule actually reflects a very "Western" dialect of visual language that I would claim is even stronger in European comics than American comics (though I don't have data to support that). In two separate studies (first, second) I coded comics from America and Japan for how many characters appeared in each panel (the second breaks down American books into Indy and Mainstream genres).

Overall, I found that American comics used far more panels showing multiple interacting characters than Japanese manga, which used overwhelmingly more panels of single characters or close ups. This would support that American books use more sequences following "Rule #1" than Japanese books.

This difference has an impact on comprehension. Being provided with only parts of a scene (single characters) forces you to infer the larger scene. This requires more machinery in the narrative grammar (what I call "Environmental-Conjunction"), i.e., the rules in people's heads that allows them to comprehend sequential images. Yet, this does not necessarily lead to poor comprehension. Rather, it simply reflects a different grammar along with the need for a different type of fluency. Neither is better or worse. Just different.

So, as described in my last post (and the comments), the problem isn't that sequences like this are "incomprehensible" in some "universal" sense, but rather that those that have difficulty with them either 1) lack fluency in this grammar, or 2) have a different set of patterns in their heads from being fluent in a different visual language (such as European VL vs. American VL vs. Japanese VL).

Nevertheless, for Campbell's purposes of whether this would help an inexperienced (i.e., non-fluent) reader: There is no data at this point suggesting that the "Western" way leads to easier comprehension. I wouldn't doubt that this might be the case though, because it forces less inference.

However, it is worth also considering that were an actual author of comics to change their dialect in this way, it may have an effect on experienced readers. I would bet that doing as Campbell suggests would actually have an adverse effect on the reading experience for a Japanese manga reader, and possibly for a reader of mainstream American comics. So, for an author considering "changing their dialect" to that of Campbell's, they may have to weigh these issues (and for which audience is intended).

Rule 2: Ordering of speech balloons and Rule #3: Speech balloons should follow a system that can be intuited and doesn’t need to be explained.

Campbell claims that "After reading the contents of one balloon, the eye is likely to go to the next nearest balloon, even if that balloon is in another panel and the eye has not yet taken in all the balloons in the current panel. "

Some eye-tracking studies give us insight on this...

First, one study found that balloon position did have an impact on how often people skipped over content. They found that panels were skipped if they followed a panel that had a balloon with a dense amount of text. Breaking apart that balloon into smaller balloons with less text lead to less skipping over panels. This implies that alterations to balloons can have an impact on reading behavior.

Another study compared the eye-movements of "novice" and "expert" readers as they navigated through comic pages. The inexperienced comic reader had erratic eye-movements across a page and focused much more on the text. In contrast, the experienced comic reader had a very deliberate order of reading, and focused on the images much more than the text. This implies that an experienced reader would not jump around to whatever balloons are closest, but an inexperienced reader might.

Thus, in this case Campbell's rules might, as is their aim, help an inexperienced reader.

Rule #4: Timing only exists in comics if the reader agrees to play the game.
Unless I am mistaken, this rule has to do with people who read ahead in a book so that crucial information is known before it's read in the narrative. This has little to do with the structure of the narrative, and has to do with people flipping through a book beforehand. Contrary to Campbell's claim, I don't see how it's any different with skipping ahead in a novel except that images show you content. 

I'd be curious for Campbell to expand here on just how authors should prevent readers from skipping ahead. The example he gives is fairly constrained and clever for preventing people from getting too much information by inadvertently reading ahead. How would this work for something like his example of "Magneto [showing] up surprisingly on the last page"?

It's worth mentioning here that how we interact with a comic as a physical object (flipping pages, accidentally looking ahead to the next page, etc) is different from our actual comprehension (the patterns in our heads that allow us to understand/produce sequential images). I don't think that Campbell confuses this issue, but it's a point worth remembering.

Rule #5: In a visual medium, a thing does not exist unless it is seen to exist
This rule applies to characters off-panel and indicated with the tail of a balloon, or to implied aspects of a depiction. Campbell notes in his example that we are to assume a character has his arms tied behind him because we don't see them (and it matches the context of the page), and that this is something that should be depicted. 
However, this rule applies also to the legs of the characters: No character on that page is depicted below the waist, yet Campbell doesn't have an issue about that except to say that having a panel where the whole figures shown (feet and all) would help provide good spacing (i.e., Rule #10: have a panel with feet on every page). He doesn't say necessarily that if not shown, we won't believe they have feet.
It is certainly the case that "undepicted" elements are part of the conventional grammar of panel framing. Studies of children show the ability to treat a panel as a "window" on a scene is correlated with experience reading comics. Comic industry lore also tells of interactions with indigenous people who did not read comics who wonder why figures in panels without legs "had no legs at all." Unless you fully lack some type of basic fluency in the visual language, then this shouldn't be an issue. 
Nevertheless, there is some validity in providing at least some notice of an element in a panel and not leaving too much to be inferred. People do track elements and concepts across panels. An element even subtly depicted once can then pervade inferentially across a sequence.

In contrast, another constraint on sequences aims at reducing how many elements need to be tracked across panels. Including too much information can lead to overload in working memory and can adversely impact comprehension. Thus, balancing these issues—what should or should not be shown—can be a delicate issue.
So, Campbell's rules cover a wide range of issues. In most cases though, he is correct that inexperienced readers may have trouble with these issues. It is not clear though whether following such rules would help those people further comprehend sequences or if some of these alterations may have adverse ramifications on the reading experience of people who are actually fluent already. However, experiments could easily test these ideas...

Friday, February 22, 2013

Fluency and dialects in understanding comics

In a recent article at The Comics Journal, Eddie Campbell describes the challenge facing some people who "can't understand comics," and offers "rules of comprehension" to help aid readers along in their ability to read comics. I thought the piece was really interesting, and Campbell intuitively taps into many ideas that are fundamental to sequential image understanding.

Nevertheless, there are several issues here being conflated that it might be nice to tease apart. We can categorize a lack of easy understanding of sequential images to two main things: 

1. Lack of visual language fluency

There legitimately are people who cannot understand sequences of images. These people mostly have never had any experience reading comics, and the most compelling I have found are people from non-Westernized countries or have never learned any language (such as deaf individuals who never learned a sign language). These are far from the impoverished conditions that Campbell is talking about, but a lack of exposure to comics would still lead someone to not being able to make sense of sequences of images at all. Not just "be confused," but legitimately have a hard time connecting the meanings between panels.

Even among "fluent" readers of comics, comprehension differs based on experience. These differences can be seen in how people move their eyes across pages, and my studies show differences in brainwave amplitudes correlated with comic reading expertise. While Campbell's rules may aid "non-fluent" people a little, the entire task of reading comics will be difficult for them at a fundamental level. 

In contrast, many of the "rules" that Campbell describes are more akin to prescriptive "rules" of "proper English" than the underlying structure of the language. These would be akin to "don't end sentences with prepositions" or "don't split infinitives"—both of which are not actual rules of English grammar. You wouldn't learn these rules in a class on English as a second language.

Rather, real rules of English would be things like "don't put adjectives after nouns" or "don't flip nominative and accusative case when using pronouns" and other rules that significantly impair the structure of a sequence. These rules are rarely produced by fluent English speakers because your mental grammar constrains the language intuitively enough to disallow them. 

Similarly, there are significant rules of the grammar of sequential images that can really impair comprehension, and these types of rules are what people who "can't understand comics" really struggle with. Yet, they are rarely violated by authors of comics, who are fluent in their visual language already. Those who are not fluent, on the other hand, lack these core rules of understanding. I plan to discuss Campbell's actual rules in my next blog post.

2. Competition with another visual language grammar

A second type of difficulty in comprehension comes from preferences we have for one type of system over another. Campbell nicely acknowledges this, calling it an "idiom" or "style," and even making the comparison to accents. Accents are the right comparison, but, unlike Campbell's belief about "idioms," accents aren't a choice. They are reflections of the patterns in people's head that they acquire from their language. Furthermore, this may be what leads to distaste in other people's systems.

For example, speakers of one dialect of English (let's say Texan) might grate on the ears of speakers of another dialect (let's say New Yorkers)…and vice versa. This doesn't mean either group lacks fluency in their language, nor are they speaking some degraded form of "pure" English (which does not exist). They simply have patterns in their brains for their languages that differ in certain features, though they are still mutually intelligible. Of course, systems become even more difficult when they are not mutually intelligible—such as English speakers (of any type) and Japanese speakers.

Different "dialects" of visual language work the same. What some may view as incomprehensible storytelling may simply be competition of one visual language grammar (let's say "Indy" comics) with another (let's say "mainstream" comics). To a reader from one camp, it may seem as though the author is "bad" at storytelling or lacks the ability to be a decent visual writer. However, it may just be that the patterns in their head is different than those of the author. They might "speak" different visual languages. 

I personally think this accounts for many of the complaints people have made about 1) comics from different countries (ex: people who have trouble with/dislike manga or bande desinée) or 2) younger artists (ex: the old guard's critique in the 90s with many Image Comics storytelling). This latter case simply is an instance of "those kids today are ruining the language," while the former is a cross-cultural reaction to a different narrative grammar.


Nevertheless, the overall idea that there is a fluency to sequential image comprehension—and some people lack it—is just what we would expect from the notion of a visual language. This idea underlies a very different perspective than Campbell's. Rather than believe that sequential images are somehow universal—and thus the problem in understanding is simply certain surface features—the idea of a visual language acknowledges that the production and comprehension of sequential images directly ties to patterns in people's heads. Because of this, despite the mutual intelligibility that iconic drawings offer, these "rules" need to be learned, and they may differ depending on which visual language you read and draw.

Thursday, February 14, 2013

The Principle of Equivalence

An overarching theme across my research is the idea that the structure and cognition of drawing and sequential images is comparable to that of language. I have tried to formalize this notion under the umbrella of the "Principle of Equivalence":

The human mind/brain treats all modalities in an equal way, given modality specific constraints.
There are two parts to this Principle. The first part (in the first clause) is the idea that the mind/brain has general cognitive principles that all factor into different domains in similar ways. We should therefore expect that the cognition of spoken language, sign language, and visual language be similar, just as we might expect facets of actions, music, dance, and others to share parts of that same cognitive orientation.

Among the things that might be similar across domains might be: 1) the ability to store systematic parts into our long term memory (i.e., a "lexicon"), 2) the ability to manipulate those parts to make larger units, 3) the hierarchical organization of sequences that enables those parts to be organized in an infinite number of ways, etc.

The second part of this Principle (the second clause) wards off an overextension of these similarities. In some ways, we should absolutely expect that different human behaviors are processed differently. However, we would expect that the nature of those differences is a direct result of the nature of the behavior itself.

For example, we might expect spoken language to differ from drawings in certain ways, because drawings are analog and spatial, while spoken language is digital and temporally constrained. All of these differences are directly related to the fact that drawings are visual-graphic while spoken language is verbal-auditory. The differences come directly from the nature of the expressions.

Overall, this Principle is affirming to the general processes of the mind. Why should the brain create lots of unique diverse ways to handle different behaviors when it can efficiently make use of various general underlying structures (like those listed above) in a variety of capacities?

Coming back to the overall idea of drawings being structured, processed, and learned comparable to language, the Principle of Equivalence demands a counter to any theory going against it. It's notable that this is the vast majority of theories about drawing from the past century, which do not think drawings are structured in a systematic, conventionalized way, but rather that they represent perception.

To these theories, they must address the key question: Why should drawing and sequential images NOT be processed like language or other human behaviors? The same questions should be asked of language and other behaviors: What makes it alone unique and different, and why is that advantageous to cognition or behavior?