Friday, February 22, 2013

Fluency and dialects in understanding comics


In a recent article at The Comics Journal, Eddie Campbell describes the challenge facing some people who "can't understand comics," and offers "rules of comprehension" to help aid readers along in their ability to read comics. I thought the piece was really interesting, and Campbell intuitively taps into many ideas that are fundamental to sequential image understanding.

Nevertheless, there are several issues here being conflated that it might be nice to tease apart. We can categorize a lack of easy understanding of sequential images to two main things: 

1. Lack of visual language fluency

There legitimately are people who cannot understand sequences of images. These people mostly have never had any experience reading comics, and the most compelling I have found are people from non-Westernized countries or have never learned any language (such as deaf individuals who never learned a sign language). These are far from the impoverished conditions that Campbell is talking about, but a lack of exposure to comics would still lead someone to not being able to make sense of sequences of images at all. Not just "be confused," but legitimately have a hard time connecting the meanings between panels.

Even among "fluent" readers of comics, comprehension differs based on experience. These differences can be seen in how people move their eyes across pages, and my studies show differences in brainwave amplitudes correlated with comic reading expertise. While Campbell's rules may aid "non-fluent" people a little, the entire task of reading comics will be difficult for them at a fundamental level. 

In contrast, many of the "rules" that Campbell describes are more akin to prescriptive "rules" of "proper English" than the underlying structure of the language. These would be akin to "don't end sentences with prepositions" or "don't split infinitives"—both of which are not actual rules of English grammar. You wouldn't learn these rules in a class on English as a second language.

Rather, real rules of English would be things like "don't put adjectives after nouns" or "don't flip nominative and accusative case when using pronouns" and other rules that significantly impair the structure of a sequence. These rules are rarely produced by fluent English speakers because your mental grammar constrains the language intuitively enough to disallow them. 

Similarly, there are significant rules of the grammar of sequential images that can really impair comprehension, and these types of rules are what people who "can't understand comics" really struggle with. Yet, they are rarely violated by authors of comics, who are fluent in their visual language already. Those who are not fluent, on the other hand, lack these core rules of understanding. I plan to discuss Campbell's actual rules in my next blog post.

2. Competition with another visual language grammar

A second type of difficulty in comprehension comes from preferences we have for one type of system over another. Campbell nicely acknowledges this, calling it an "idiom" or "style," and even making the comparison to accents. Accents are the right comparison, but, unlike Campbell's belief about "idioms," accents aren't a choice. They are reflections of the patterns in people's head that they acquire from their language. Furthermore, this may be what leads to distaste in other people's systems.

For example, speakers of one dialect of English (let's say Texan) might grate on the ears of speakers of another dialect (let's say New Yorkers)…and vice versa. This doesn't mean either group lacks fluency in their language, nor are they speaking some degraded form of "pure" English (which does not exist). They simply have patterns in their brains for their languages that differ in certain features, though they are still mutually intelligible. Of course, systems become even more difficult when they are not mutually intelligible—such as English speakers (of any type) and Japanese speakers.

Different "dialects" of visual language work the same. What some may view as incomprehensible storytelling may simply be competition of one visual language grammar (let's say "Indy" comics) with another (let's say "mainstream" comics). To a reader from one camp, it may seem as though the author is "bad" at storytelling or lacks the ability to be a decent visual writer. However, it may just be that the patterns in their head is different than those of the author. They might "speak" different visual languages. 

I personally think this accounts for many of the complaints people have made about 1) comics from different countries (ex: people who have trouble with/dislike manga or bande desinée) or 2) younger artists (ex: the old guard's critique in the 90s with many Image Comics storytelling). This latter case simply is an instance of "those kids today are ruining the language," while the former is a cross-cultural reaction to a different narrative grammar.

--------

Nevertheless, the overall idea that there is a fluency to sequential image comprehension—and some people lack it—is just what we would expect from the notion of a visual language. This idea underlies a very different perspective than Campbell's. Rather than believe that sequential images are somehow universal—and thus the problem in understanding is simply certain surface features—the idea of a visual language acknowledges that the production and comprehension of sequential images directly ties to patterns in people's heads. Because of this, despite the mutual intelligibility that iconic drawings offer, these "rules" need to be learned, and they may differ depending on which visual language you read and draw.

8 comments:

Eddie Campbell said...

I made comments at the other place where this is posted. Since you have ignored them there, I'll cut and paste them here.
You're probably not intending to be argumentative, but here we go...

"and offers “rules of comprehension” to help aid readers"

you misrepresented my argument right there in your first sentence. It's about helping artists 'speak' more clearly. I said that at the outset. It's 'rules toward a rhetoric of comics.' That's clearly for the benefit of the speaker, not the hearer, yes?

"Because of this, despite the mutual intelligibility that iconic drawings offer, these “rules” need to be learned, and they may differ depending on which visual language you read and draw."

The 'rules' were for artisans making the work, why would the people reading them need to learn anything? A person listening to a song doesn't need to know about musical tonality. The intent of the rules is to lessen the amount of foreknowledge that a reader needs when coming to a comic. If you're saying that we'd need a lot more rules I would have to agree. List yours and let's get on with it.


"unlike Campbell’s belief about “idioms,” accents aren’t a choice."

ummm... Once you are made aware that your way of speaking is restricting your accessibility to a larger audience, you can CHOOSE to speak in a more universal way than the one you grew up with. I know lots of people who have deliberately suppressed or changed an accent, including my sister.

"Rather, real rules of English would be things like “don’t put adjectives after nouns” or “don’t flip nominative and accusative case when using pronouns” and other rules that significantly impair the structure of a sequence."

surely the rule about 'not putting word balloons at the bottoms of panels when they are likely to confuse the reading order' fits the 'real rules of English' analogy. Did you speed-read my essay?

"While Campbell’s rules may aid “non-fluent” people a little, the entire task of reading comics will be difficult for them at a fundamental level."

Yes, I agree, comics are not the best way to communicate with non-fluent readers. But we're well outside of my argument now, so i'll leave that one.

Eddie Campbell

Eddie Campbell said...

"the impoverished conditions that Campbell is talking about,"

Every time my eye wanders back over your piece I see other things that make no sense. Every person I referred to who couldn't read comics is an educated person (eg. "one or two people who were so well brought up that they had never read a comic."). Where did you get 'impoverished'. It suddenly occurs to me that another Eddie Campbell must have posted an essay simultaneously and that you have confused the two of us.

It occurs to me furthermore that your assistant must have written your piece and posted in your absence. When you get back from holiday, let me know what you think ;)

Neil Cohn said...

Eddie, thank you for your replies and clarifications. I didn't know about another posting of my article (this feed is taken up other places), so my apologies for not replying there. And really, my assistants have a lot of science to do that keeps them far too busy to reply to your comments. ;-)

I wasn't trying to be argumentative, just to tease out some of your insights with regard to my theories of the science behind comic comprehension. Common themes in my work are 1) rules of comprehension, and 2) why some people cannot understand sequential images. It only seems natural to reply to your article with what actual research tells us on the topic.

-----

I realize the difference between rules intended for rhetoric and those intended for comprehension. This was much of my motivation for my post. Rules of rhetoric are about tailoring an expression in a particular way, while the "rules of comprehension" that I'm talking about are the basic building blocks of understanding something cognitively.

My point was not to say that you are somehow miscategorizing your rhetorical rules for the cognitive ones. Rather, I was trying to express that people who don't have the cognitive rules may not benefit a ton from the rhetorical ones taken by "artisans." (this is an empirical question of course, which could be tested using experiments).

Unfortunately, listing "my rules" would take a lot more than a blog post! They are scattered amongst several of my articles at this point, though they'll be covered in part in my book due out next year. Someday I do hope to bridge the scholarship/praxis gap though and do a book on how my cognitive theories can be used for rhetorical purposes. When I do it, I'd love to hear your feedback!

(And rhetoric isn't just for the speaker. It's for the speaker to be clear to the hearer. If not for the hearer, why would the speaker care what they sound like? Isn't that actually your point?)

Neil Cohn said...


Yes, one can deliberately change accents to a more socially dominant accent (no language is "universal" or more correct). My point here was to say that many conventions in comics do not result from a lack of artistry, but because they are learned parts of people's conventions.

Now, I fully accept the argument that people could potentially change their "visual accents" using the types of rules you suggest—absolutely! However, I don't think we we should confuse following these rules as becoming something more "universal" or that they strip someone of an "accent." Rather, they simply teach people how to have a different visual accent that is also conventional.

This point actually makes your rules even more interesting, since they can be seen as a reflection of how one visual dialect sees another (at least for some traits... I'm thinking of #1 in particular).

This speaks to one of my biggest points with my post: Sometimes perceived "inadequacies" are merely reflections of people speaking a different dialect than you.

------

I'm not so sure about placement of balloons being analogous to a "real" rule of English. Certainly, putting a balloon at a panel's bottom would make it confusing if other balloons were above it that needed to be read after it.

However, your example was putting a balloon at the bottom and having it connect to a balloon in a different panel that would be several panels away. I'm not sure that would really have an impact on a "fluent" reader. My prediction would be that a panel border would provide too strong of a boundary for that to have any impact. If that were true, it would support that bottom-panel balloon placement is not too strong a "real rule," though it might be a nice rhetorical one.

On that note, in the spirit of your claims: you're probably right that balloon placement could help an inexperienced reader. In my next post I'll actually describe a study that supports this.

------

By "impoverished" I meant "impoverished in their ability to understand comics." It has nothing to do with education, and I realize now that might be jargon from psycholinguisitics that sounds odd out of context.

------

Anyhow, thanks again for your comments! Later this week I plan to post a breakdown of each of your rules and discuss what science tells us about each. Hopefully you'll comment on that one too.

Eddie Campbell said...

"My point here was to say that many conventions in comics do not result from a lack of artistry, but because they are learned parts of people's conventions.

yes, that's what I already said. I was very careful about that. I made it clear that I was using as an example the work of a prize winning writer-artist.

"Sometimes perceived "inadequacies" are merely reflections of people speaking a different dialect than you.. "

sure, but Queen's English is easier to follow than Glaswegian. If you're speaking on the world stage, you don't want to be talking Glaswegian. I made it clear that comic book conventions are fine if you're only interested in reaching a comic book readership.

"My prediction would be that a panel border would provide too strong of a boundary for that to have any impact."

I had already made it clear that the panel boundaries in this case were not distinct: "and it takes a moment to figure out that the middle row of panels is not so much inset into the large upper panel as that all of them are set against a black ground, with slight overlaps."

By "impoverished" I meant "impoverished in their ability to understand comics."

And would you say that a person who does not understand Glaswegian or Brooklynese or cockney, is 'impoverished' in this ability? Or would you pick another word, seeing how this would likely lead to confusion, given that the person's misunderstanding is possibly due to their being from a higher socio-economic class.

Hey, Neil, I admire your work and your ideas. I'm sure you know I always have done. But I write my things like legal documents. Most of the arguments that might come up I've already anticipated. Respond accordingly.

Eddie

Eddie Campbell said...

ps. one other thing.

"(And rhetoric isn't just for the speaker. It's for the speaker to be clear to the hearer. If not for the hearer, why would the speaker care what they sound like? Isn't that actually your point?)"

This appears to overlook that the possibility that the speaker may wish to mislead the hearer, such as to persuade him to an action that he would normally avoid, as in a political speech, or persuading men to join the army. Or as I put it in the other set of comments, when I was emphasizing that rhetoric presupposes that speaker and hearer speak the same language:
"In the foreign field you might use a big stick; rhetoric is what you use to justify it when you get home."

Eddie Campbell said...

otherwise, I'm not sure why we're arguing. I always thought we were facing in the same direction. perhaps my use of the word 'comprehension' has triggered a crossfire.

Let's see what you say in your breakdown then, with regard to whether my guidelines lead to greater clarity for the fluent, the non-fluent or nobody.

Eddie

Neil Cohn said...

Eddie, I know you've enjoyed my work, as I have yours. I do think ultimately we're facing the same direction.

As I said, I didn't intend this to be argumentative, nor was the piece intended to attack anything you said, just supplement and explain in different ways. Like a "good little scholar," I feel debate is worthwhile for growing ideas, and it need not be taken either personally nor adversarially (though it unfortunately often requires a bumpy process of figuring out each other's jargon).

Just to let things roll along, I'll let your statements stand and you can address my next post where I'll break down what the scientific research says about your rules. (A preview: it's not entirely antithetical to your ideas)


(An unrelated aside: I have a book about my research due out early next year. My publisher will likely be looking for people to offer quotes, meaning they'll likely get an advance look at it. Would you like to be included on that list? If so, shoot me email!)