Wednesday, August 26, 2009

Comics reading: Competence and performance

Often times when I give talks, especially concerning layout or the visual grammar of sequences of images, one of the questions inevitably says something along the lines of "But, there's no guarantee that the reader will view a page in the proper order." The high variability of possible choices or readings of a comic page makes it hard for them to accept a steadfast theory of comprehension.

However, a similar issue was at play in linguistics back in the 1950s, and was one that Noam Chomsky importantly addressed in his distinction between competence and performance. Competence refers to the (idealized) organization of rules and constraints in our minds that guides us to understand language. Performance is the vast variability that happens in real life exchanges.

For example, someone might say something like this over the phone:

I ...uh... I went *cough cough* to the store *STATIC***--oday and, like, saw *CAR HORN*--ohn from my class in the check-*hiccup*-out line.

There are lots of interruptions, unclear portions and distractions. However, most likely a listener would glean from this a sentence like:

I went to the store today and saw John from my class in the checkout line.

The rules in your head are not bothered by the messiness of the context — your attentional system can filter out a lot of it.

The same is true of reading a comic page. Let's say you start in one panel and go to another, then realize it shouldn't have gone next. You're not belying the mental rules that go into comprehension — in fact, those rules are what tell you it's the wrong order. These actual rules of comprehension are unconscious to your awareness.

Your (unconscious) competence wins out over the messiness and variability of performance.

This same issue may be at play with comparisons of comics to film. Yes, film and comics are presented differently (one static, one moving), but that doesn't necessarily mean that their comprehension in people's minds is entirely different. The difference in presentation may be a "performance" issue, while the comprehension is a "competence" issue. (Though, in my mind there is bound to be at least some variance due to that presentation difference — motion vs. static — which will need experiments to explore... yay science!).

I should point out also that, in linguistics, there are some debates over the complete reality of this split in notions. For example, for a long time it was argued that words like "um" and "uh" are just performance clutter. However, research has shown that these actually hold meaning for the discourse (essentially signaling how long a pause the speaker is going to make before continuing to talk).

Nevertheless, for many issues facing the comprehension of "comics" (and/or film), it is an important split to make.

Thursday, August 20, 2009

Abstract Comics and Visual Language

Abstract Comics is a new collection of non-representational comics by a variety of authors, including my esteemed blogging colleague Derik Badman. Besides being a beautifully done work of artistry and imagination, among particular crowds it spurs the question "If these are comics, then what 'are comics'?"

To this end, the book (and the nature of the contents in general) makes a good conversation piece. They also make a great example of how to distinguish the difference between "Comics" and my notion of "Visual Language", which is made clearest by teasing out just what parts of cognition these comics engage.

First, we should ask the question "Are abstract comics of this sort 'comics' at all?" I would have to say "YES" — simply because they call them so. However, they are not instances of visual language.

To repeat my theory... Visual language is a system of patterns (from people's heads) in that expresses concepts through the graphic modality using sequential images. So, visual language uses three interlocking cognitive systems:

1. Graphic modality
2. Meaning
3. Sequential structure (i.e. grammar)

Like spoken or signed languages, this system is culturally relative, meaning that different cultures use different visual languages (for example, "standard" manga style versus "standard" superhero style dialects), and this system is used socioculturally in comics paired along with written languages.

Comics are written in visual language (± text) the same way that novels are written in English. Novels aren't English, and Comics aren't visual language. (This equation would essentially be McCloud's position, that comics "are" sequential images).

Given this, abstract comics most definitely are comics — because they call themselves comics, they are formatted like comics, they are made by people affiliated with comics, sold in comic stores, etc. They satisfy most all the sociocultural aspects that one would expect comics to fulfill.

However, they do not use visual language. They don't use representational depictions that reflect patterns in people's heads. They don't seem to have any sort of grammar of narrative structure. They don't depict any meanings at all. In other words, they use just one of visual language's structures from our cognitive system*:

1. Graphic modality
2. Meaning
3. Sequential structure (i.e. grammar)**

They are merely playing with the graphic modality in a sequential way that entirely lacks meaning (in the conceptual sense, not necessarily the "artsy" sense).

As a result, abstract comics make a great example of comics (and art) that lack visual language. This is the inverse of something like an airplane safety card, which is representational, but lies outside the sociocultural category of calling it "comics." I should say also, that this isn't a bad thing — it's quite fun, clever, and creative, and further goes to my point that these notions of "comics" and "visual language" are separate.

* Interestingly, the one cognitive structure they do use though, is visual language's navigational system for how to move through a page layout. In many ways this is an ancillary system to the primary VL system for expressing meanings in sequence, but it is curious that this seems to be one of the formal ways in which abstract comics get to call themselves "comics".

** I haven't analyzed the pieces in the book enough, but it is not inconceivable that graphic sequences with no meaning could still retain a narrative grammar. For example, Action Star substitution incorporated into a sequence with purely visual surrounding panels could lend to this result.

Tuesday, August 11, 2009

Memory, Experience, and Comics Comprehension

In my last post, I discussed some traits of this quote by Chris Ware found from this blog post

“I don't like to think of my work as 'cinematic.' A movie is passive -- you're watching it, taking it in. Where a comic strip, it's completely active: you have to read it, search it for meaning, for the connection with your entire experience and your memory. Yes, you do have the illusion of watching something happen in a comic strip -- but if it's done well, it comes alive on the page like a novel. A novel is the most interactive thing ever created.”

The other thing I find interesting about this quite is that I have a hard time believing that people "imagine" things while reading comics that connect with their "entire experience" and "memory." There are two things that this quote implies:

1) That people are converting their reading experience into consciously clear interpretations (imagery, sounds, etc) while reading a comic (a notion that echoes McCloud's Closure).

2) That people's creation of meaning is entirely based on experience ("Empiricism").

Concerning the first point, I know when I read a comic, I don't necessarily feel like I "fill in" any missing imagery with mental imagery of my own. I don't visualize anything that isn't in the pages. I do understand it, and make the mental connections between and across images/words, but there is no additional imagery added. Novels do create this imagery (for some but not all people) because it isn't provided already.

This blog post has replied to my earlier posting expressing that Ware's meaning of "active" comprehension relates to this sort of filling in of sensory information that's missing. Again, I am hesitant to accept that people are actually imagining sounds, smells, motion, etc. while reading a comic.

Novels certainly allow people to create visual imagery — but vision is our primary sensory modality, so I find it unsurprising that this would happen. I'm less confident about the other senses.

SO....If you actually do feel like you create mental imagery while reading comics, I want to hear about it in the comments please!

On the second point, there is quite a lot of evidence that our understanding of meaning does not necessarily come from experience (and certainly not conscious experience). That's not to say all of it is innate, but there's a give and take between innate meaning and acquired meaning — the debate is over the percentages.

What I'd be more confident stating though is that when reading a comic, I doubt people are actively referencing overt memories or experiences in order to comprehend a sequence. Rather, they are drawing upon their abstract concepts — just like when they read a book, or yes, see a movie.

Sunday, August 02, 2009

How active is comic comprehension versus film?

Dash Shaw writes an interesting post delving into the "cinematic" nature of comics that explores thoughts from authors like Chris Ware with many insightful quotes.

Relevant to some of this discussion might be that some believe comics to have predated the film techniques. Or, the idea that this is a competence versus performance issue — that film uses the same mental structures as comics, just with a different presentation (this will be a topic of an upcoming post).

Indeed, in several experiments of mine I show comic panels one after another, one at a time where the participants have no control over the pacing. My participants have no difficulty understanding these or accepting them "as comics" (no one has ever questioned the labeling).

Most interesting though is this quote of Ware's from the post:
“I don't like to think of my work as 'cinematic.' A movie is passive -- you're watching it, taking it in. Where a comic strip, it's completely active: you have to read it, search it for meaning, for the connection with your entire experience and your memory. Yes, you do have the illusion of watching something happen in a comic strip -- but if it's done well, it comes alive on the page like a novel. A novel is the most interactive thing ever created.”

I have a lot of responses to this quote, but I'll save some for a later post. Right now, I want to question what "watching it, taking it in" means with regard to film comprehension that's different than the "active" comprehension of comics. This is a common thread in comparisions, so I wonder whether Ware (and many others who also do it) is conflating the presentation of a comic/film versus its comprehension.

Is the sense that film is "less active" because it's pace of viewing is not controlled by the viewer? This to me seems like a trivial thing in terms of comprehension. The process of understanding (i.e. piecing together the meaning between images, words, and/or sound) should maintain roughly the same.

If comprehension were different, we would expect grossly different results if we presented the same comic strips in different ways in an experiment (that could use any number of measures of comprehension). Let's say we had three different methods:

1) a comic page where all panels were laid out in a grid, possibly controlled so that subsequent panels only appear when a button is pressed by the reader ("self paced reading")
2) a "self-paced reading" task where only one panel is on a screen at a time
3) a presentation with no participant control, where only one panel appears on a screen at a time for a designated amount of time

Now, I would expect no significant difference in the ability of people to comprehend these different scenarios. This is all about presentation, not the content of the strips, since those could stay the same across all of these (and other) presentation methods.

#3 on this list is essentially the same type of presentation that film uses. Granted, I will wholeheartedly agree, film's use of *moving* images certainly does change comprehension. However, there still has to be meaningful connections between and across film shots (be it live-action or animated). These would be of the same "active" sort of connections that Ware describes. Indeed, you can replace film shots for panels in the above three options and probably get the same sort of comprehension as you would for static comics. So, instead of issues of presentation, the focus of questions should instead be on issues of comprehension, like:

How does the comprehension of static versus moving sequential images differ?

How does moving images within a unit (shot vs. panel) change its comprehension?

How does the use of moving across a scene (as in panning, zooming, etc.) differ in comprehension from it's static presentation in panels (or shots)?

AND... we can't really answer these questions without an adequate theory of how comprehension of sequential images works in the first place, which is essentially what my research for the past several years has been focused on.