Tuesday, May 26, 2009

What is "Visual Language"? Video Talk

A few weeks ago I had the pleasure of giving a talk at the University of Toronto about my theories of visual language hosted by the Knowledge Media Design Institute. They have now posted this talk online to be viewed in full (beware, it's quite long) on their website.

If anyone is really interested in just what my theory entails overall, this is definitely worth watching. It lays out the basic principles and issues for what exactly I mean by "visual language", and how that relates to "comics", "language", "art", etc.

Note: Be forewarned that the slides they show do not have their full proper animations, so might end up looking a bit more cluttered than is intended.

Saturday, May 23, 2009

Cartoony vs. Realistic Images in the Brain

Choi, Yeojeong, Kim, Takhwan, & Jaeseung Jeong. 2008. "EEG Source Localization during Empathy of Iconic and Realistic Cartoon Characters." Organization for Human Brain Mapping (OHBM), Melbourne, Australia ,15-19 June, 2008

In McCloud's Understanding Comics he proposed his theory of "cartoon identification" that cartoony* images are "identified" with better than realistic images. This study (pdf) tested McCloud's theory by using behavioral measures of a 7-point rating and EEG measures of the brain's electrical activity.

I've found that this theory of McCloud's was a bit ambiguous, since people have interpreted it in two different ways. It can either mean that people "identify" with cartoony images meaning...

1) They are perceived cognitively at a more "base" level.
2) That they empathize with the characters more.

Critics have usually tapped into the second reading, since it is close to a claim about how people "identify" with characters in a literary sense. However, I've always been more partial to the first interpretation (which I attempted to codify further in my book), though the two views aren't necessarily mutually exclusive. Indeed, cartoony images could evoke more empathy because they are more conceptually basic. This study aimed to examine the second version, strictly with a view of the degree of "empathy" styles create.

The experimenters contrasted comic strips that featured two opposing characters who were depicted in either realistic or cartoony styles (as in the example above) and put into different scenarios to evoke reader sympathy via who wins the confrontation (strips had varying endings, ex. winner is congratulated by a woman vs. loser is consoled by a woman). Participants only viewed one depiction, and only one option for the ending.

They then compared ratings in a behavioral study on a 7-point scale measuring empathy to both "winners" and "losers", and in a separate population, measured EEG brainwaves for the same stimuli.

Brain areas were activated that related to social perception, recognizing facial expressions, and seeing another person's pain**. They found both higher behavioral ratings of empathy and greater activation in the brain for areas for the cartoony characters than the realistic characters for both "winners" and "losers" (though different brain areas for different roles). They take these results to be support for McCloud's theory of identification that indeed, cartoony images do invoke greater empathy from a reader than realistic images.


*The authors, and McCloud, often use "iconic" to mean "cartoony" — I'm going to avoid this because it doesn't accurately convey what "iconic" means in a semiotic sense (i.e. meaning through resemblance). Technically, both cartoony and realistic images are "iconic."

** Just a caveat for those who actually follow the link to the pdf poster. The study shows nice pretty pictures of brains with activated regions to support its hypothesis, but these can be misleading given the actual methods used. Unlike a technique like fMRI, EEG does not give much information about where in the brain something occurs, and is much better at when it appears (i.e. the timing of processing). These brain images and results were gained using a "source localization" procedure which extrapolated from the data what brain regions were being used, a technique that is commonly employed, but often controversial. This isn't to say that the results are inaccurate, but they should be understood with this context.

Tuesday, May 19, 2009

Consistent reading time for comic pages?

I had a link sent to me recently asking about this blog post that claims all comic pages regardless of content are read at 3.75 seconds per page. A relevant section:

I read in one of Frederik Schodt’s excellent books on manga that a study concluded that readers spend an average of 3.75 seconds on a comic page. My own observations of myself and others has led me to believe that time frame to be fairly consistent, by which I mean not dependent on the contents of a page. Unless a writer really creates an absolutely confounding monologue or an artist completely botches an integral sequence, readers do not seem to change their flipping speed for “difficult,” wordy, nor beautiful pages. This yields somewhat counter-intuitive results, in my estimation. Single panel pages, which should ostensibly be flown through, allow one image to be lingered on or “drunk in” because that one drawing is granted the full 3.75 seconds. Pages with many panels, taken to the extreme above, should require a slower, more contemplative pace. But they do not. They seem to clock at the same 3.75, meaning the eyes need to whip through these images to make it in time.

He then goes on to advocate different strategies of layout based on the idea that readers will go through it at this magic time of 3.75 seconds. Since I wrote a lengthy counter-rebuttal to this claim, I figured I ought to post it here too.

According to the science I've seen, this does not seem to be the case. The amount of time people spend on each individual panel varies based on how much information is in it, it's order in the sequence, as well as possibly size of both panel and page, and a whole page time varies definitely the way the page layout is organized.

From a very general study of my own relating to times it takes 4 panel comic strips to be read, I found each panel at an average of 1.5 seconds per panel when readers press a button to advance through panels. But, it does vary per position and narrative structure — first panels are consistently slower, panels after major events much slower. However, if you just take that average and multiply it by 4, that gives you 6 seconds for one 4 panel Peanuts strip that has no words in it.

In my last study, I found reading times varying between .6 and 1.8 seconds per panel (small times for panels that had very little information, such as blank panels or those with just action stars), with the full 6 panel strips clocking in around 6 ±2.5 seconds.

Plus, the *uncited* study that was mentioned in the blog is for manga (and if I recall correctly, Schodt also doesn't cite the actual study), which consistently 1) use slightly less panels per page (my corpus study — "Cross Cultural Space" — showed both American and Japanese books to have 5 panels per page, but manga had a lower standard deviation), and 2) use less balloons per page. Furthermore, eye-tracking studies show that fluent readers skip over far more balloons than non-fluent readers — so, less balloons means less reading time, especially for fluent readers.

The poster here then says that he finds this time to be consistent to his own experience — but you can't know such a thing from anecdotal evidence. You would have to have measures to substantiate it.

And, even if it were true that on average pages are read at a pace of 3.75 seconds — which, I imagine there is some average time out there if one were to crunch all the numbers — there is no way that we would feel the need to allot different time to different panels based on some intuitive feeling that we "want" to read each page in a specified amount of time.

Rather, the time it takes to read a page all depends on its content and the fluency of the reader.

Sunday, May 17, 2009

Congrats to my visual linguists!

I'm very excited to say a big hearty Congratulations to my research assistant Natalie, who graduated today from Tufts. She put in an incredible year's worth of work in the preparation stages of a major project of mine, apexing in the online study that many of you all took (thanks from both of us!). Last week she presented a poster of the results in a department-wide session, and did a fantastic job.

Also, I'm pleased to say that the students of my Visual Linguistics of Comics class did a great job throughout this semester, and gave me great food for thought, as well as encouragement that this is only the first of many such courses.

Wednesday, May 06, 2009

Public Lecture in Toronto, May 14th

For those of you in the Toronto area, I will be giving the first talk in the Visual Thinking lecture series at the University of Toronto on Thursday, May 14th at 5pm. The talk is open to the public and more information can be found here. Here's the topic of my talk:

What is "Visual Language"?: What Comics can Tell Us About the Mind

Many theories describing "visual language" have been emerging from diverse fields including computer science, communications, and design. However, often these approaches rely on metaphoric or folk notions of "language" without delving deeper into what Language actually consists of, especially on a cognitive level. This talk will present Visual Language Theory from the view of the linguistic and cognitive sciences to discuss what "language" entails, and thereby exploring just what it means to have a literal theory of a graphic modality of language. The result will be a view of graphic communication and the capacity for drawing that is embedded alongside other mental capacities and divorced from socio-cultural labels that stymie its recognition.

For those of you not in Toronto, I've been told that this presentation will be recorded and available on iTunes following the event. I'll post more on this as I get more info.