Thursday, December 24, 2009

Almost back...

Okay, one more status update then we're back to some real posts...

My brainwave study has nicely concluded and I've now moved on to analyzing the results. Things look fantastic, so I'm very excited about working to get these written up and submitted to a journal in the coming months. This whole project has been a very long one: 1.5 years making stimuli, 7 months running the experiments, probably another 3 or so writing it up. It'll be a relief once its done (and the next beginning), but the results seem to be worth it...

Otherwise, I've finally had some time to tinker with some decent blog posts. So, I'm just going to let them accrue until after the holiday weekend then come back and start posting again. It's been a busy busy few months, but I think some more blogging will be back in the mix shortly. Phew!

Monday, November 30, 2009

Not MIA!

Unfortunately blogging has had to take somewhat of a back seat lately, as my workload has been just crazy.

At the end of this week I should finish running participants through my brainwave study using comics, though I can already confirm that the results are just fantastic. I think I can confidently characterize how the brain processes narrative sequential images, and its really quite exciting (with pretty solid evidence against panel transitions). Stay tuned... (though I'll say already that this will be the topic of my ComicCon talk in summer).

Otherwise, I've been working on writing up several other studies that have been ruminating in my computer. What with the multiple articles currently under review for journals and books, over the next few years I should have a steady stream of new papers emerging.

Beyond that, the publisher De Gruyter has been nice enough to provide me with a review copy of the recent volume Multimodal Metaphor which has lots of chapters on comics. Once I have a little more time I'll be writing reviews of the book and relevant chapters.

Sunday, November 15, 2009

Transition Overload!

I've frequently heard it said that every panel in a comic has to connect to every other panel. I've tried to go about showing the problems with individual transitions or McCloud's closure, but I have yet to tap into this issue on the blog.

Potentially, this could be at least somewhat the notion behind Groensteen's ideas of braiding and arthrology. "Restrained arthrology" says there are meaningful connections made between all juxtaposed panel relationships (i.e. what McCloud would call panel transitions), while "general arthrology" pushes this up to possible connections between all panels in a book ("braiding").

In my book, I toyed with a similar idea of multi-connected transitions for very specific examples, but cast it aside before proposing my alternative approach based on Chomsky's generative grammar. However, the "every panel with every other" viewpoints are far more unconstrained than my approach ever was.

One of the biggest problems with this "every panel with every other" as a theory of comprehension is that it would just overwhelm a person's working memory to keep that many things active in their head with no guiding structures. So, I figured it would be worth the exercise of showing how ridiculous such an assertion might be...

For an average book that has 6 panels per page for 24 pages, this would give 144 panels in a book. Connections between any two panels in those 144 would be calculable as 144!/(2!•142!). This would build up to 10,296 possible transitions as every possible combination would additively create with each successive panel read, as the mind continuously retained them all in memory. Granted, not all panel relationships might need to establish an explicit "transition", but all connections would be necessary to at least confirm or deny the need for an explicit transition.

Without any underlying structure to guide such connections, this would be overwhelming for human memory to handle. Rather, there needs to be something explicit provided by the mind to manage (and group/subdivide) such connections— just like a grammar for language. Transitions and general principles of "arthrology" just won't do it.

Wednesday, October 28, 2009

Learning to read your brain(waves)

So, today marks a minor milestone for me, as I ran my very first study of comics looking at people's brainwaves. The image here to the right is from that first participant, and each of the lines is of a different type of sequence that we are experimentally testing.

So, what does this tell us?

Absolutely nothing.


Data from one participant doesn't say much, but give me a few more weeks and these waves will be (hopefully) showing interesting information about how the brain processes sequences of images.

Tuesday, October 20, 2009

Review: Adventures in Cartooning

Adventures in Cartooning is a fun and creative book by James Sturm and two of his graduates from the Center for Cartoon Studies, Andrew Arnold and Alexis Frederick-Frost, and published by First-Second.

It is designed as a how-to create comics book, though the lessons almost wholly come in narrative form as the Magic Cartooning Elf and other characters discuss the properties of comic creation while carrying out a simple and fun story. The book is aimed largely at children, and the humor reflects it (though did make me laugh aloud at parts — particularly the "Warning" on the back cover, which is just the sort of thing to get kids to pick it up).

Most of the overt instruction is fairly simple — things like what is a panel, how text can enrich images, orders of word bubbles, and the nature of different graphic devices like motion lines or dotted panel borders. The last several pages of the book also contain sections on cartooning basics that make explicit several of the lessons as well as some additional instruction.

However, because most of the instruction comes narratively, there ends up being only a limited amount of things instructed. This is a shame, because several techniques the authors use are very clever, elegant, and well worth instructing learners if they don't notice them.

For example, in one section the characters climb and then descend a mountain. On the climb, the three small square panels of the page are positioned climbing upwards left-to-right so that the line of the mountain-side is retained, while the opposite configuration occurs for the descent. On another page the characters sink down into water, with the length of the panel growing away from the top of the page in each panel to show falling deeper and deeper.

These are fun, simple, and effective techniques that comic creators can put to great use, though without the explicit instruction I fear they might be lost by less observant readers. Perhaps allowing for some additional non-narrative instruction would allow for even more of these aspects to be brought out explicitly. It would be easy to do this with the little labels and arrows used on the cover (which appear nowhere else in the book), or with the Elf character hovering outside the panels to point things out as well.

(On the flipside, I can understand why the authors might feel they just want to give kids the basics and not overwhelm them with too many concepts, though I'm inclined to think kids can handle it).

Additionally, I greatly liked how much of the book was carried out without text and the implications that text is only used to enrich visuals. This subtly reinforces the development of the visual language grammar in learning — which is no doubt the intent.

Overall, the book is a good read and would likely be a useful tool for helping young creators get on their way to creating graphic stories.

Thursday, October 08, 2009

Review: The Power of Comics

Duncan, Randy & Matthew J. Smith. 2009. The Power of Comics. New York: Continuum Books

The Power of Comics is a recently released “first textbook ever” for “comics studies”, authored by Randy Duncan and Matthew J. Smith (book website here).

Perhaps to be expected from a book on the general “comic studies”, it includes a broad range of topics, from the history of comic books to comprehension of the medium, to creators and fandom. In many ways, the vastly disparate topics brings into question the overall utility of conceiving of a singular field called “comic studies.” Indeed, a person interested in one particular subtopic may find the entire rest of the book irrelevant. No doubt though, the scope of the book is meant to be inclusive and to at least cast the possibility that such a broad field could exist.

As a textbook, it succeeds in content, scope, and execution. The chapters are well laid out, have thoughtful questions at their ends, and several chapters end with very practical examples of analysis that serve as models for students. Chapters also reference a broad range of experts (discussed below), which further validates breadth and depth of this growing scholarship.

Given the nature of my own interests, I’ll focus primarily on the chapters dissecting the medium, Chapters 6 and 7. These chapters on formal analysis largely expand and refine Duncan’s earlier papers on “Comic Book Communication.” The theories stick largely to the expected status quo of theory: nothing overly radical or surprising jumps out, which is perhaps to be expected from a textbook.

In some ways though, this is a detriment. Despite growing works on formal aspects of the “comic medium”, most of these chapters rely on concepts inspired by Eisner and McCloud, supplemented by Groensteen and scattered others. However, much of these ideas go by with little regard for debates to their legitimacy. For example, “closure” is assumed to be true and never questioned as being valid at all (though multiple interpretations are presented). Others include the (erroneous) belief that the “gutter” somehow contributes to meaning, and the idea that each panel must be connected with every other panel in comprehension (can you say “working memory overload”?).

The primary focus of these chapters describe aspects of meaning-making, providing summaries of overview notions that intertwine across numerous levels of comprehension (sequence, layout, etc). The chapters are chock full of information, much of it useful to a beginner and some likely useful to more advanced students. I particularly liked the idea of an interplay between the reduction and expansion of information in the medium as a nice simple way to describe the status quo of considerations about the medium.

However, on the whole, numerous broad theoretical concepts are discussed without much real theoretical grist to them. Again, this might not be bad since the format is a textbook — elaboration on numerous topics would be impossible for the space.

Nevertheless, some aspects describing theories are a bit roughshot — for example describing “icon/index/symbol signs” could have benefited from better explanations, and proper terminology (it should either drop “signs” or be “iconic/indexical/symbolic signs”) and attribution to their originator (Charles Sanders Peirce) would at make for a good mention that these theories are roughly 100 years old.

Additionally, while McCloud and, at least somewhat, Eisner, are recognized for their theoretical insights and contribution to the canon, the realms of theory and praxis get blurred further in the chapters with quotes from numerous comic creators and an unheard of “comics art collector,” which among the few experts seems curious as legitimate sources. It also brings into question just what and who these chapters are aiming at: Theory? Analysis? Praxis?

Despite its limitations, The Power of Comics marks an accurate state of the field (whatever it might be) for studying comics. For good or bad, the theories in Chapters 6 and 7 reflect a particular paradigm of thinking about the medium. While it is my personal belief that formal theories have moved into a more sophisticated state, the views expressed in this book reflect what will someday be viewed as a nascent growth stage of considering the medium. For that, it almost seems like a “living history”, saying where we’ve been while knowing bigger things have and will appear.

Overall though, the book — including the theory chapters — is reasonably good for a “first textbook on comics,” and I would imagine it will fast become a standard text for those sorts of classes.

Wednesday, September 30, 2009

Panels connected by sequentiality

Derik has a short post that makes a nice note about how understanding of individual panels is sometimes conditioned by their context in a sequence.

I think this is a very important point that is well illustrated by his example. Sometimes, understanding of the elements in an individual panel relies on the information in other panels.

Most all cultures and individuals have little trouble decoding most propositional information in images (i.e. that an image of a horse means "a horse", or that an image of person is "a person", etc). However, certain individuals may have trouble comprehending the objects if their meaning is conditioned by a sequence. For example, this sort of meaning by context is often what children under four and other "non-visual language fluent" readers (or those fluent in a different type of system) struggle with.

Why is this important/interesting?

1) It lends validity to the idea that there is a fluency required for sequential image comprehension (and thus that there is a "system" guiding understanding to be fluent in).

2) It implies that even perceptual understanding (i.e. vision and object recognition) might rely on sequential understanding in these contexts, meaning that mere perception alone isn't enough to explain sequential image comprehension (i.e. again, a system for sequential images is necessary).

3) It hints that these sequential images were created to be in sequence and not just as random images strewn together. This is also a support against an image-to-image system of understanding like panel transitions, since transitions could function no matter what is thrown next to each other. This sort of execution has a more global scope: it's a whole sequence made to be a whole sequence, not just one after another.

Monday, September 28, 2009

Review: Metaphor and Metonymy in Comics Storytelling

Kukkonen, Karin. 2008. Beyond Language: Metaphor and Metonymy in Comics Storytelling. English Language Notes 46 (2):89-98.

This paper from the literature point of view explores meaning-making in comics, particularly from metonymy and metaphor. It argues that the "semiotic" approaches of European comics scholarship that dissect parts into structrualist "minimal units" are insufficient to capture the complexity of comics' meanings, and is thereby a tacit argument against viewing "comics as a language" in the semiotic sense.

(Groensteen takes this same perspective against minimal units, though maintains the "comics as a language equation. I actually think that "minimal units" are *kind of* there, but it's beside the point, since linguistics hasn't really been concerned about "minimal units" since around the 1950s...)

While she does explain and support the cognitive linguistics view of metaphor taken from Lakoff and Turner, she does not actually use it in exposition. Most of the examples of metaphor and metonymy she cites are through a close reading of Watchmen, involving large scale metaphors on the scale of plots, themes, and motifs, and doesn't ever cite the correspondences of one "coneptual domain to another" that conceptual metaphor entails.

Her view of metonymy is equally broad. For part-whole metonymy she cites scenes where a whole understanding of an environment is given across multiple panels. This would imply that all instances where multiple characters are shown in their own panels but part of the same broader environment (what I call "Environmental-Conjunction") are metonymic, because they construct a broader whole by only seeing the parts. This is a curious proposition that I (mostly) like, though one that seems at least partially limited by not having a robust view on the broader narrative grammar for how sequential images are comprehended.

Friday, September 18, 2009

Japanese children, drawing, and imitation

These are a couple great articles by Brent Wilson about how children in different cultures learn to draw, particularly contrasting the Japanese with other cultures. Both articles contain more extensive discussions that I'll mention here, only focusing on some of the highlights...

Wilson, Brent. (1999). Becoming Japanese: Manga, Children’s Drawings, and the Construction of National Character. Visual Arts Research, 25(2), 48-60.

Here Wilson provides fairly striking evidence that over two-thirds of the drawings produced by Japanese children in primary education (K-6) are imitative of manga. However, even many of his "non-manga" types could have been drawn from manga and still might be drawn in that style. He also notes a developmental trajectory: There is a decline in non-manga drawings after kindergarten.

I would guess that this is in part a socialization process. Since kids at kindergarten start playing with other children in a structured arena, and all of them know manga, there is more of a motivation to draw in the style of the group than whatever way they might be drawing at home. Also, it reflects a growing sense of literacy. Since kids are learning to read, manga become reading material in addition to stimuli for learning to draw.

Wilson, B. (1988). The Artistic Tower of Babel: Inextricable Links Between Culture and Graphic Development. In G. W. Hardiman & T. Zernich (Eds.), Discerning Art: Concepts and Issues. Champaign, IL: Stipes Publishing Company.

In this article, Wilson focuses on the argument against the belief that the Romantic view that children develop an inborn "artistic" capacity and that external influences are bad for it. He compares this in part to a sort of "Tower of Babel" phenomenon, where everyone has a universal inborn "language of art" that develops uniquely suited to each individual.

His theory against these claims is that graphic symbolism is a language that is transmitted through cultural patterns imitatively. He cites numerous evidence for this from numerous countries and time periods.

Pertaining to Japanese children though, he has a fascinating note that nearly all of the 6-year old Japanese children they studied could draw coherent sequential stories, compared with other countries where only about half of some groups of 12-year olds could. He also notes that Japanese children use increasingly more complex methods of graphic narrative as they age, (examples from 9 to 12), though they are all imitative of manga techniques.

These examples support that drawing ability is learned, as well as that imitation boosts abilities rather than hampers it.

Friday, September 11, 2009

Leigh's Paintings

As long as I'm recharging a bit for future blogging and managing my preparation for the new semester here at school, I felt I should at least devote one blog post to shill for the new website my father established for his paintings. He's put up numerous works of his along with essays about them, including this one below which hangs in my home.

Most of my family is highly creative in various domains. So, if you're interested in checking out some nice impressionist-ish paintings, go check it out!

Sunday, September 06, 2009

Blogging MIA

I have unfortunately been remiss in blogging lately. I actually just got back from an intense 8-day long Soo Bahk Do martial art test (not to mention the month and a half intense training preparing for it), and am just now reintegrating into "normal life" (if you can call what I do "normal")...

This week is the start of the Tufts school year, which will excitingly feature my teaching of a class on "Comics and the Mind" as well as numerous research projects. Hopefully I'll have some more experiments online soon, as well as papers released for public consumption. In the meantime, stay tuned and I hope to resume regular blogging soon...

Wednesday, August 26, 2009

Comics reading: Competence and performance

Often times when I give talks, especially concerning layout or the visual grammar of sequences of images, one of the questions inevitably says something along the lines of "But, there's no guarantee that the reader will view a page in the proper order." The high variability of possible choices or readings of a comic page makes it hard for them to accept a steadfast theory of comprehension.

However, a similar issue was at play in linguistics back in the 1950s, and was one that Noam Chomsky importantly addressed in his distinction between competence and performance. Competence refers to the (idealized) organization of rules and constraints in our minds that guides us to understand language. Performance is the vast variability that happens in real life exchanges.

For example, someone might say something like this over the phone:

I ...uh... I went *cough cough* to the store *STATIC***--oday and, like, saw *CAR HORN*--ohn from my class in the check-*hiccup*-out line.

There are lots of interruptions, unclear portions and distractions. However, most likely a listener would glean from this a sentence like:

I went to the store today and saw John from my class in the checkout line.

The rules in your head are not bothered by the messiness of the context — your attentional system can filter out a lot of it.

The same is true of reading a comic page. Let's say you start in one panel and go to another, then realize it shouldn't have gone next. You're not belying the mental rules that go into comprehension — in fact, those rules are what tell you it's the wrong order. These actual rules of comprehension are unconscious to your awareness.

Your (unconscious) competence wins out over the messiness and variability of performance.

This same issue may be at play with comparisons of comics to film. Yes, film and comics are presented differently (one static, one moving), but that doesn't necessarily mean that their comprehension in people's minds is entirely different. The difference in presentation may be a "performance" issue, while the comprehension is a "competence" issue. (Though, in my mind there is bound to be at least some variance due to that presentation difference — motion vs. static — which will need experiments to explore... yay science!).

I should point out also that, in linguistics, there are some debates over the complete reality of this split in notions. For example, for a long time it was argued that words like "um" and "uh" are just performance clutter. However, research has shown that these actually hold meaning for the discourse (essentially signaling how long a pause the speaker is going to make before continuing to talk).

Nevertheless, for many issues facing the comprehension of "comics" (and/or film), it is an important split to make.

Thursday, August 20, 2009

Abstract Comics and Visual Language

Abstract Comics is a new collection of non-representational comics by a variety of authors, including my esteemed blogging colleague Derik Badman. Besides being a beautifully done work of artistry and imagination, among particular crowds it spurs the question "If these are comics, then what 'are comics'?"

To this end, the book (and the nature of the contents in general) makes a good conversation piece. They also make a great example of how to distinguish the difference between "Comics" and my notion of "Visual Language", which is made clearest by teasing out just what parts of cognition these comics engage.

First, we should ask the question "Are abstract comics of this sort 'comics' at all?" I would have to say "YES" — simply because they call them so. However, they are not instances of visual language.

To repeat my theory... Visual language is a system of patterns (from people's heads) in that expresses concepts through the graphic modality using sequential images. So, visual language uses three interlocking cognitive systems:

1. Graphic modality
2. Meaning
3. Sequential structure (i.e. grammar)

Like spoken or signed languages, this system is culturally relative, meaning that different cultures use different visual languages (for example, "standard" manga style versus "standard" superhero style dialects), and this system is used socioculturally in comics paired along with written languages.

Comics are written in visual language (± text) the same way that novels are written in English. Novels aren't English, and Comics aren't visual language. (This equation would essentially be McCloud's position, that comics "are" sequential images).

Given this, abstract comics most definitely are comics — because they call themselves comics, they are formatted like comics, they are made by people affiliated with comics, sold in comic stores, etc. They satisfy most all the sociocultural aspects that one would expect comics to fulfill.

However, they do not use visual language. They don't use representational depictions that reflect patterns in people's heads. They don't seem to have any sort of grammar of narrative structure. They don't depict any meanings at all. In other words, they use just one of visual language's structures from our cognitive system*:

1. Graphic modality
2. Meaning
3. Sequential structure (i.e. grammar)**

They are merely playing with the graphic modality in a sequential way that entirely lacks meaning (in the conceptual sense, not necessarily the "artsy" sense).

As a result, abstract comics make a great example of comics (and art) that lack visual language. This is the inverse of something like an airplane safety card, which is representational, but lies outside the sociocultural category of calling it "comics." I should say also, that this isn't a bad thing — it's quite fun, clever, and creative, and further goes to my point that these notions of "comics" and "visual language" are separate.

* Interestingly, the one cognitive structure they do use though, is visual language's navigational system for how to move through a page layout. In many ways this is an ancillary system to the primary VL system for expressing meanings in sequence, but it is curious that this seems to be one of the formal ways in which abstract comics get to call themselves "comics".

** I haven't analyzed the pieces in the book enough, but it is not inconceivable that graphic sequences with no meaning could still retain a narrative grammar. For example, Action Star substitution incorporated into a sequence with purely visual surrounding panels could lend to this result.

Tuesday, August 11, 2009

Memory, Experience, and Comics Comprehension

In my last post, I discussed some traits of this quote by Chris Ware found from this blog post

“I don't like to think of my work as 'cinematic.' A movie is passive -- you're watching it, taking it in. Where a comic strip, it's completely active: you have to read it, search it for meaning, for the connection with your entire experience and your memory. Yes, you do have the illusion of watching something happen in a comic strip -- but if it's done well, it comes alive on the page like a novel. A novel is the most interactive thing ever created.”

The other thing I find interesting about this quite is that I have a hard time believing that people "imagine" things while reading comics that connect with their "entire experience" and "memory." There are two things that this quote implies:

1) That people are converting their reading experience into consciously clear interpretations (imagery, sounds, etc) while reading a comic (a notion that echoes McCloud's Closure).

2) That people's creation of meaning is entirely based on experience ("Empiricism").

Concerning the first point, I know when I read a comic, I don't necessarily feel like I "fill in" any missing imagery with mental imagery of my own. I don't visualize anything that isn't in the pages. I do understand it, and make the mental connections between and across images/words, but there is no additional imagery added. Novels do create this imagery (for some but not all people) because it isn't provided already.

This blog post has replied to my earlier posting expressing that Ware's meaning of "active" comprehension relates to this sort of filling in of sensory information that's missing. Again, I am hesitant to accept that people are actually imagining sounds, smells, motion, etc. while reading a comic.

Novels certainly allow people to create visual imagery — but vision is our primary sensory modality, so I find it unsurprising that this would happen. I'm less confident about the other senses.

SO....If you actually do feel like you create mental imagery while reading comics, I want to hear about it in the comments please!

On the second point, there is quite a lot of evidence that our understanding of meaning does not necessarily come from experience (and certainly not conscious experience). That's not to say all of it is innate, but there's a give and take between innate meaning and acquired meaning — the debate is over the percentages.

What I'd be more confident stating though is that when reading a comic, I doubt people are actively referencing overt memories or experiences in order to comprehend a sequence. Rather, they are drawing upon their abstract concepts — just like when they read a book, or yes, see a movie.

Sunday, August 02, 2009

How active is comic comprehension versus film?

Dash Shaw writes an interesting post delving into the "cinematic" nature of comics that explores thoughts from authors like Chris Ware with many insightful quotes.

Relevant to some of this discussion might be that some believe comics to have predated the film techniques. Or, the idea that this is a competence versus performance issue — that film uses the same mental structures as comics, just with a different presentation (this will be a topic of an upcoming post).

Indeed, in several experiments of mine I show comic panels one after another, one at a time where the participants have no control over the pacing. My participants have no difficulty understanding these or accepting them "as comics" (no one has ever questioned the labeling).

Most interesting though is this quote of Ware's from the post:
“I don't like to think of my work as 'cinematic.' A movie is passive -- you're watching it, taking it in. Where a comic strip, it's completely active: you have to read it, search it for meaning, for the connection with your entire experience and your memory. Yes, you do have the illusion of watching something happen in a comic strip -- but if it's done well, it comes alive on the page like a novel. A novel is the most interactive thing ever created.”

I have a lot of responses to this quote, but I'll save some for a later post. Right now, I want to question what "watching it, taking it in" means with regard to film comprehension that's different than the "active" comprehension of comics. This is a common thread in comparisions, so I wonder whether Ware (and many others who also do it) is conflating the presentation of a comic/film versus its comprehension.

Is the sense that film is "less active" because it's pace of viewing is not controlled by the viewer? This to me seems like a trivial thing in terms of comprehension. The process of understanding (i.e. piecing together the meaning between images, words, and/or sound) should maintain roughly the same.

If comprehension were different, we would expect grossly different results if we presented the same comic strips in different ways in an experiment (that could use any number of measures of comprehension). Let's say we had three different methods:

1) a comic page where all panels were laid out in a grid, possibly controlled so that subsequent panels only appear when a button is pressed by the reader ("self paced reading")
2) a "self-paced reading" task where only one panel is on a screen at a time
3) a presentation with no participant control, where only one panel appears on a screen at a time for a designated amount of time

Now, I would expect no significant difference in the ability of people to comprehend these different scenarios. This is all about presentation, not the content of the strips, since those could stay the same across all of these (and other) presentation methods.

#3 on this list is essentially the same type of presentation that film uses. Granted, I will wholeheartedly agree, film's use of *moving* images certainly does change comprehension. However, there still has to be meaningful connections between and across film shots (be it live-action or animated). These would be of the same "active" sort of connections that Ware describes. Indeed, you can replace film shots for panels in the above three options and probably get the same sort of comprehension as you would for static comics. So, instead of issues of presentation, the focus of questions should instead be on issues of comprehension, like:

How does the comprehension of static versus moving sequential images differ?

How does moving images within a unit (shot vs. panel) change its comprehension?

How does the use of moving across a scene (as in panning, zooming, etc.) differ in comprehension from it's static presentation in panels (or shots)?

AND... we can't really answer these questions without an adequate theory of how comprehension of sequential images works in the first place, which is essentially what my research for the past several years has been focused on.

Monday, July 20, 2009

(^_^) ... Emoticons and the Brain

Masahide Yuasa, Keiichi Saito,Naoki Mukawa. 2006. Emoticons convey emotions without cognition of faces: an fMRI study. Conference on Human Factors in Computing Systems. April 22-27, 2006. Montréal, Québec, Canada

Related to the previous post on a testing of McCloud's "Cartoon Identification Theory" on cartoony vs. realistic images in the brain, here's a study using fMRI (brain scans) to look at emoticons which at this point are perhaps the most simplified signs for faces we use.

Faces have been the focus of a lot of debate in cognitive neuroscience, particularly about the "face area" in the brain. One side says it's an area strictly devoted to processing human faces, the other side says that it's an "expertise" area and it activates because humans are experts at recognizing faces. It is one of the most fiery debates in cognitive neuroscience, and learned about in most all intro classes.

Amazingly, this study shows no activation of this "face area" when looking at emoticons. :-O

Using fMRI, the authors compare Asian style emoticons (non rotated) with averaged faces (photos of multiple faces that have been blended to be more "generic") that were expressing the emotions of happiness and sadness. Emoticons appeared first on their own, and in a second study embedded within sentences, while non-emoticon signs using the same characters were also used as fillers (i.e. ":O*-<").

They found that photos of faces activate both areas pertaining to emotional valence (right inferior frontal gyrus) and facial recognition (right fusiform gyrus), while emoticons only activate emotional areas but not face areas. That is, as the authors say, "Remarkably, emoticons convey emotions without cognition of faces."

This finding has very interesting consequences for understanding how brains process varying degrees of complexity in images. The implication here at least is that more simplified faces become tied more explicitly to a "symbolic" meaning as opposed to their iconic meaning of resembling what they look like. That is, more simplified images strip down the meaning to its core meaning disconnected to the iconic reference that they are framed within.

It would be interesting to see a graded approach to this — such as taking different degrees of representation from McCloud's gradient of "cartoonification" (or to use my term, "haplosis"). Are there different degrees of activation for different representations? Does activation for the fusiform gyrus all suddenly drop off at a certain level of simplification? How does this affect the debate that the fusiform gyrus is an "expertise" area rather than a face area?


Tuesday, July 14, 2009

Comic-Con 09: Action Stars!

I am excited to say that my panel at Comic-Con next week should be a fun one. I'll be on the Comics Theory panel in room 30AB on Thursday at 11:30. Here's the official description:

11:30-1:00 Comics Arts Conference Session #2: Comics Theory— Dru H. Jeffries (Concordia University) argues that Zack Snyder’s film 300 mimics the form of comics by manipulating film styles, particularly slow motion. Neil Cohn (Tufts University) presents the results of psychology experiments investigating how our minds make sense of the sequence of images in comics. David B. Olsen (St. Louis University) uses examples that include Winsor McCay, Paul Pope, and Alan Moore to demonstrate how we negotiate the rhythm of comics reading. Room 30AB

More specifically, I'll be presenting my studies on "Action Stars", which I first posted last Fall. If you took my online experiment earlier this year where you had to fill in the blank describing various strips, you'll now get to learn what that was all about. I'm really excited about this presentation, so come on out!

Tuesday, July 07, 2009

WGBH Boston: One Guest Interview

A few months back I recorded an interview with WGBH Boston that has finally come online:

They went with the "not just for kids" lines a bit, but it was a quick tantalizer into my work I thought. The most challenging thing I thought was how fast it was. We had a number of topics we wanted to hit, and only 5 minutes to do it with no editing. So, that meant everything went by very fast. Enjoy!

Thursday, June 25, 2009

Comics and Film: A Narrative Perspective

Christiansen, Hans-Christian. 2000. Comics and Film: A Narrative Perspective. In Comics and Culture: Analytical and Theoretical Approaches to Comics, edited by A. Magnussen and H.-C. Christiansen. Copenhagen: Museum of Tusculanum Press.

This paper attempts to draw from film theory to inform the understanding of the comic medium. He discusses things like film shots/cuts, etc. especially in light of Bordwell and Thompson's work claiming that comprehension runs along a continuum of:

Conventions --> Norms --> Cross-cultural universals --> Deep structure of visual storytelling

For example, he claims that if knowledge of "cinematographic narration" comes from perceptual understandings, then it dispels the myth that these techniques were "invented" by early innovators of cinema. Rather, then would involve universal "deep structures of visual storytelling" grounded in perception.

Of course, this would assume that "cinematographic" techniques don't do anything that is out of the realm of perceptual knowledge, which isn't always the case — for example, with showing zooms alternating with panels of various characters ("Refiner Projection") and external settings that provide superordinate place information.

He also makes a case for complexity of storytelling matching genre, and says "as a rule, there is a higher frequency of point of view-structures in adventure comics than in romance comics" though provides no evidence for this claim.

While film uses movement, comics use static images. Thus, he breaks down aspects of temporal continuity between panels into three filmic types of cuts:

1) Matched cuts - where a movement is continued from one panel to another
2) Movement images - continuity created by action and shot-to-shot closure
3) Elliptical cuts - a discontinuous relation between shots that requires greater inference to understand the relationship

He argues that comics primarily use the final type of elliptical cuts the most, and that it is not disruptive in comics because of their layout on a page. He closes this appeal by stating that this process of ellipsis is actually a part of McCloud's notion of "amplification of simplification" for representing visual events, and that the way in which comics are understood may lie at the root of how film is understood at a base level (which I'd mostly agree with).

Tuesday, June 16, 2009

7 Years!

Oop, looks like I forgot to post a couple weeks ago on 5/31 that this website hit its 7th birthday! Whoohooo  — belated celebration!

Not much time to blog these days. I'm currently running an extensive experiment that has gone through numerous rounds of troubleshooting, so that's occupying most of my time. If this design works though, I'll be dancing around campus in amazement. I'm also finally writing up the results of a study that was run about two years ago, which is a nice relief.

Plus, I'm preparing my Comic-Con talk for this year that's on experiments I ran based on this blog post about "Action Stars." I'll have more up on that as the date grows nearer.

As usual though, if there's any question or issue that you'd like me to blog about, I'm always willing to take requests.

Tuesday, May 26, 2009

What is "Visual Language"? Video Talk

A few weeks ago I had the pleasure of giving a talk at the University of Toronto about my theories of visual language hosted by the Knowledge Media Design Institute. They have now posted this talk online to be viewed in full (beware, it's quite long) on their website.

If anyone is really interested in just what my theory entails overall, this is definitely worth watching. It lays out the basic principles and issues for what exactly I mean by "visual language", and how that relates to "comics", "language", "art", etc.

Note: Be forewarned that the slides they show do not have their full proper animations, so might end up looking a bit more cluttered than is intended.

Saturday, May 23, 2009

Cartoony vs. Realistic Images in the Brain

Choi, Yeojeong, Kim, Takhwan, & Jaeseung Jeong. 2008. "EEG Source Localization during Empathy of Iconic and Realistic Cartoon Characters." Organization for Human Brain Mapping (OHBM), Melbourne, Australia ,15-19 June, 2008

In McCloud's Understanding Comics he proposed his theory of "cartoon identification" that cartoony* images are "identified" with better than realistic images. This study (pdf) tested McCloud's theory by using behavioral measures of a 7-point rating and EEG measures of the brain's electrical activity.

I've found that this theory of McCloud's was a bit ambiguous, since people have interpreted it in two different ways. It can either mean that people "identify" with cartoony images meaning...

1) They are perceived cognitively at a more "base" level.
2) That they empathize with the characters more.

Critics have usually tapped into the second reading, since it is close to a claim about how people "identify" with characters in a literary sense. However, I've always been more partial to the first interpretation (which I attempted to codify further in my book), though the two views aren't necessarily mutually exclusive. Indeed, cartoony images could evoke more empathy because they are more conceptually basic. This study aimed to examine the second version, strictly with a view of the degree of "empathy" styles create.

The experimenters contrasted comic strips that featured two opposing characters who were depicted in either realistic or cartoony styles (as in the example above) and put into different scenarios to evoke reader sympathy via who wins the confrontation (strips had varying endings, ex. winner is congratulated by a woman vs. loser is consoled by a woman). Participants only viewed one depiction, and only one option for the ending.

They then compared ratings in a behavioral study on a 7-point scale measuring empathy to both "winners" and "losers", and in a separate population, measured EEG brainwaves for the same stimuli.

Brain areas were activated that related to social perception, recognizing facial expressions, and seeing another person's pain**. They found both higher behavioral ratings of empathy and greater activation in the brain for areas for the cartoony characters than the realistic characters for both "winners" and "losers" (though different brain areas for different roles). They take these results to be support for McCloud's theory of identification that indeed, cartoony images do invoke greater empathy from a reader than realistic images.


*The authors, and McCloud, often use "iconic" to mean "cartoony" — I'm going to avoid this because it doesn't accurately convey what "iconic" means in a semiotic sense (i.e. meaning through resemblance). Technically, both cartoony and realistic images are "iconic."

** Just a caveat for those who actually follow the link to the pdf poster. The study shows nice pretty pictures of brains with activated regions to support its hypothesis, but these can be misleading given the actual methods used. Unlike a technique like fMRI, EEG does not give much information about where in the brain something occurs, and is much better at when it appears (i.e. the timing of processing). These brain images and results were gained using a "source localization" procedure which extrapolated from the data what brain regions were being used, a technique that is commonly employed, but often controversial. This isn't to say that the results are inaccurate, but they should be understood with this context.

Tuesday, May 19, 2009

Consistent reading time for comic pages?

I had a link sent to me recently asking about this blog post that claims all comic pages regardless of content are read at 3.75 seconds per page. A relevant section:

I read in one of Frederik Schodt’s excellent books on manga that a study concluded that readers spend an average of 3.75 seconds on a comic page. My own observations of myself and others has led me to believe that time frame to be fairly consistent, by which I mean not dependent on the contents of a page. Unless a writer really creates an absolutely confounding monologue or an artist completely botches an integral sequence, readers do not seem to change their flipping speed for “difficult,” wordy, nor beautiful pages. This yields somewhat counter-intuitive results, in my estimation. Single panel pages, which should ostensibly be flown through, allow one image to be lingered on or “drunk in” because that one drawing is granted the full 3.75 seconds. Pages with many panels, taken to the extreme above, should require a slower, more contemplative pace. But they do not. They seem to clock at the same 3.75, meaning the eyes need to whip through these images to make it in time.

He then goes on to advocate different strategies of layout based on the idea that readers will go through it at this magic time of 3.75 seconds. Since I wrote a lengthy counter-rebuttal to this claim, I figured I ought to post it here too.

According to the science I've seen, this does not seem to be the case. The amount of time people spend on each individual panel varies based on how much information is in it, it's order in the sequence, as well as possibly size of both panel and page, and a whole page time varies definitely the way the page layout is organized.

From a very general study of my own relating to times it takes 4 panel comic strips to be read, I found each panel at an average of 1.5 seconds per panel when readers press a button to advance through panels. But, it does vary per position and narrative structure — first panels are consistently slower, panels after major events much slower. However, if you just take that average and multiply it by 4, that gives you 6 seconds for one 4 panel Peanuts strip that has no words in it.

In my last study, I found reading times varying between .6 and 1.8 seconds per panel (small times for panels that had very little information, such as blank panels or those with just action stars), with the full 6 panel strips clocking in around 6 ±2.5 seconds.

Plus, the *uncited* study that was mentioned in the blog is for manga (and if I recall correctly, Schodt also doesn't cite the actual study), which consistently 1) use slightly less panels per page (my corpus study — "Cross Cultural Space" — showed both American and Japanese books to have 5 panels per page, but manga had a lower standard deviation), and 2) use less balloons per page. Furthermore, eye-tracking studies show that fluent readers skip over far more balloons than non-fluent readers — so, less balloons means less reading time, especially for fluent readers.

The poster here then says that he finds this time to be consistent to his own experience — but you can't know such a thing from anecdotal evidence. You would have to have measures to substantiate it.

And, even if it were true that on average pages are read at a pace of 3.75 seconds — which, I imagine there is some average time out there if one were to crunch all the numbers — there is no way that we would feel the need to allot different time to different panels based on some intuitive feeling that we "want" to read each page in a specified amount of time.

Rather, the time it takes to read a page all depends on its content and the fluency of the reader.

Sunday, May 17, 2009

Congrats to my visual linguists!

I'm very excited to say a big hearty Congratulations to my research assistant Natalie, who graduated today from Tufts. She put in an incredible year's worth of work in the preparation stages of a major project of mine, apexing in the online study that many of you all took (thanks from both of us!). Last week she presented a poster of the results in a department-wide session, and did a fantastic job.

Also, I'm pleased to say that the students of my Visual Linguistics of Comics class did a great job throughout this semester, and gave me great food for thought, as well as encouragement that this is only the first of many such courses.

Wednesday, May 06, 2009

Public Lecture in Toronto, May 14th

For those of you in the Toronto area, I will be giving the first talk in the Visual Thinking lecture series at the University of Toronto on Thursday, May 14th at 5pm. The talk is open to the public and more information can be found here. Here's the topic of my talk:

What is "Visual Language"?: What Comics can Tell Us About the Mind

Many theories describing "visual language" have been emerging from diverse fields including computer science, communications, and design. However, often these approaches rely on metaphoric or folk notions of "language" without delving deeper into what Language actually consists of, especially on a cognitive level. This talk will present Visual Language Theory from the view of the linguistic and cognitive sciences to discuss what "language" entails, and thereby exploring just what it means to have a literal theory of a graphic modality of language. The result will be a view of graphic communication and the capacity for drawing that is embedded alongside other mental capacities and divorced from socio-cultural labels that stymie its recognition.

For those of you not in Toronto, I've been told that this presentation will be recorded and available on iTunes following the event. I'll post more on this as I get more info.

Thursday, April 23, 2009

Take my Comic Experiment!

Hey everybody, I once again have another comics related experiment online and would be extremely grateful for your help!

This experiment consists of making judgements about various created comic strips, and takes roughly 20 minutes (though at first glance you may think it looks long, trust me, it doesn't take much time). Participation enters you into a drawing for a $50 Gift Certificate to Best Buy (either online or in store).

This project has a deadline of early next week (Midnight on Tues, 4/28), so your help sooner rather than later would be most appreciated.

****UPDATE: This project has been completed. Thanks very much to everyone who participated! Stay tuned for more experiments in the future, or email me to be contacted directly about future experiments.******

The study can be found online HERE.

Thanks very much!

Oh, and by the way, to all who participated in my last online study (thanks again!), a presentation of that overall project will likely be given as my talk at this year's ComicCon...

Tuesday, March 31, 2009

Indexing Events with Panels

A comment on my review of Magnussen's piece on semiotics in comics asked me to expand on this part:
More interestingly, she claims that the "still-images of actions" are also indexical, because they only show a part of a broader temporal whole action. This is probably the most astute observation of the piece, yet receives relatively little relative attention. This is a key insight, and would be worth expanding on.

First of all, in the semiotics of C.S. Peirce, indexicality is a means through which reference is garnered via causation or indication. For example, an index finger that points to something doesn't mean that thing, it indicates the thing has meaning. The finger is just saying "for the real meaning look over there." Also, if I saw a footprint in the sand, it indexes the person who once walked there, because of the causation stepping there created.

Another aspect to indexicality can happen through part-whole relationships. By showing just a hand, you index the rest of the body (assuming it hasn't been detached....*shudder*).

So, related to Magnussen's point, I'm recalling this particularly salient image in my mind from the book How to Draw the Marvel Way depicting a figure punching in several points throughout the overall action.

What Magnussen seems to imply is that a single snapshot of one part of this event sequence indexes the whole rest of the sequence. I would agree with this in general, though I think it's likely that different places within that progression will be more or less salient as indicating the whole.

For example, in How to Draw..., Lee and Buscema's advice is to use the maximally intense points of that sequence — the ends and beginnings of the action marked "best" or "not bad." These sections of the action seem more representative of the action than the medial parts. In semiotic terms, they would index the overall action better than the parts in the middle, which are less representative of the overall action.

Research seems to have borne out their intuition. Studies have shown that people's comprehension of events is better for the maximally preparatory and completed parts of an action, over that of the middle. In fact, even 10-11 month old infants seem to parse events through these outer boundaries.

Now, it would be unsurprising if Magnussen's statement were attempted to expand beyond the representation in single panels, out to across panels. So, let's say that two panels show both the beginning and end parts of that punch sequence. Here, there is a sense that the whole middle part of the action is indexed by seeing the ends — i.e. you know the middle happened but didn't see it.

This would be, essentially, what McCloud is arguing for with Closure. That, because we know the course of events, we "fill in" knowledge of the whole action by seeing the parts. However, even McCloud acknowledges that not all panel relations are of actions (for example, his Subject and Aspect transitions), though his notion of Closure is extended over all of them.

While I do not believe that "Closure" happens to "fill in the gutter" between panels (for numerous reasons), I do think that part of a represented action might index full actions. I don't know if I'd say that the whole action, including the middle parts, is "manifested" somehow in our minds. However, the reference of a part of an action certainly would index the concept of the whole action, by only being a sliver of it.

Monday, March 23, 2009

Review: Semiotics and comics

Magnussen, Anne. (2000). The Semiotics of C.S. Peirce as a Theoretical Framework for the Understanding of Comics. In A. Magnussen & H.-C. Christiansen (Eds.), Comics and Culture: Analytical and Theoretical Approaches to Comics. Copenhagen: Museum of Tusculanum Press. pp.193-207

Magnussen's paper attempts to apply Peircean semiotics to the comic medium, though little of the actual paper focuses on those intents. Much of the paper is concerned with the definition of "comics", which is too bad, because it is nearly irrelevant for the overall claims attempted to be made. Like others, she conflates the structural aspects of the visual medium with the definition of "comics" though, so that "comics" is a sign type built out of image parts < panel < panel sequence < comic.

She makes a parallel to Van Dijk and Kintsch's 1983 discourse approach, equating panels with propositions and citing "local coherences" between panels created through inferences (which could essentially be the same as McCloud's notion of "closure"). The "global coherence" of panels relation to the whole is interpretted through narrative schemas.

Because she conflates "comics" with stories/sequential images, she states:
"For a comic not to be a story, it should be possible to create a global coherence on the basis of something other than story-structure, and in which the local coherences are made on inferences based on parameters other than actions, actors, time and place" (198).

She then gets into the usual bind regarding whether a comic without a story is actually considered a "comic" or not. If accepting that comics ≠ sequential images, this would be of no concern.

Her main analysis focuses on the semiotic elements of the example comic. She primarily focuses on icons, indexes, and symbols, and leaves out aspects of Peirce's model regarding the nature of the "Sign vehicle" that would change many of the interpretations related to conventionality. For instance, if she had those notions, simplified icons would not be interpreted as symbolic, but just types of "legisigns".

Most of her argument though is that comics use indexical signs beyond just icons and symbols. For example, she claims that word balloons are indexical because of the attribution given by their Tails. More interestingly, she claims that the "still-images of actions" are also indexical, because they only show a part of a broader temporal whole action. This is probably the most astute observation of the piece, yet receives relatively little relative attention. This is a key insight, and would be worth expanding on.

Friday, March 13, 2009

Random Creative Links

As I'm coming up on Spring Break here at Tufts, all seems well as we (hopefully) transition into Springtime (hopefully — a Californian like me can't take much more of this cold before I go batty). Here are a bunch of random links that seem to have fallen my way:

Stricken Pot Pie has some fantastic comics made entirely from embroideries. While I can't say much for the story yet (it's still developing) the sheer idea of it is creative enough to warrant a link.

My friend Alexander Danner pointed me towards this interesting site, where the artist takes children's drawings and then paints/draws them in the same proportions with much more "realistic" representation. Some of the monsters are actually quite unnerving, but again, the idea is certainly a creative one.

Finally, if you're in the creative mood yourself, chalk up Strip Generator to the list of numerous low frills web-based comic creation sites out there.

Tuesday, March 10, 2009

Faces in comics

Tan, Ed S. 2001. The Telling Face in Comic Strip and Graphic Novel. In The Graphic Novel, edited by J. Baetens: Leuwen University Press.

This study looked at the knowledge of faces and facial expressions in comics assuming that facial expressions are universal and thus comics rely on these universal cues in representation. In the analysis of Tintin comics, consistent basic emotions are depicted, often with hyperbolic exaggeration. In comparison, the emotions in Maus are more downplayed and minimally focused upon. This is hypothesized as due to the “seriousness” of the comparative subject matter.

Ultimately it concludes (like in McCloud’s Making Comics), that comic drawing uses a small set of emotional primitives that are employed in combination for universal emotional expression. Curiously, though McCloud expresses they are universal in his book, his recent blog post on the topic implies that he thinks facial expressions are also a learned behavior.

On the whole, I'm curious what this would do with data from countries like Japan that vary from findings of universal facial expressions. Would Japanese manga also reflect the different interpretations of expressions held by their culture? Does this mean that the facial expressions manga might sometimes be misunderstood by non-Japanese reading them?

On the whole, this paper could have benefited from more systematic coding than the roughshot sampling of expressions in selected pages from a limited sample size. How many expressions were there? Of what kinds? How do people from those different populations interpret the expressions from differing countries? Etc.

Thursday, March 05, 2009

Tufts Daily Feature

The Tufts Daily, Tufts University's student newspaper, has a feature article about me and my work in today's edition, which is also readable online.

Thursday, February 26, 2009

Why Do You Make Comics write in visual language, If You Do?

Over at TalkAboutComics, Joey Manley asks "Why do people draw comics?", and makes the observation that...

A lot of people who read comics also make them. Maybe even most people.
Almost everybody I’ve ever met who reads comics has, at some point or another, actually made one, even if he or she never showed it to anybody.

To me, this has to do with the nature of the visual language of sequential images. Unfortunately, our failed definition of "comics" conflates the idea that "comics=sequential images", when really "comics=cultural context" and "sequential images=visual language." By this notion, what Joey is talking about is that comic readers are highly likely to create stories (comics) using this visual language.

Perhaps this should be unsurprising then, since it means this visual language works like any other language. People get exposure to it, and imitate it in order to learn and practice their fluency.

This is not a case of people "drawing comics" the way that filmmakers "make movies" or other types of "artistic" craftsmanship. Rather, this is more like the way that Americans "speak English" or Quebecois "parle en Français" because those are the languages of the communities to which they belong. Comic readers constitute the language group for visual language in America, so their "drawing comics" is simply participating in the (visual) language of their community.

Note that this also applies to the particular graphic dialect that they might partake in: those who read manga are likely to draw like manga, those that read superhero books are likely to draw that way.

Tuesday, February 24, 2009

More on the Burnt City Bowl

Apparently I've been getting linked to a lot for my previous post on the Burnt City animation drawing of an illustrated goat jumping along the outside of an earthenware bowl.

In my post, I argued that — contrary to the fairly dubious claims that this was the "world's earliest animation — this is better suited to be thought of as just an ancient example of visual language use. The closest modern analog would be a "comic", though I hate applying the term to ancient artifacts.

Now, I was recently revisiting the claims that this is animation based on some of the comments to the original that redid an animation based on the bowl. Admittedly, these do look better than the original. I suppose the idea would be that this bowl would be spun on a base where the "flip-book" quality would emerge.

However, what I'm wondering about is why would anyone expect that this would have such a usage? Isn't it a lot more logical just to assume that it was used as a bowl. Why would we think that ancient bowls with sequential images on the outside are somehow exempt from their normal use as receptacles for holding things?

In fact, beyond the modern coffee mugs that do this, we have lots of historical examples as well. For example, there is a whole huge collection of these types of bowls from ancient Maya.

So... why shouldn't we think that its primary function is just as a bowl, and that it just happens to have a cool example of sequential images on it with absolutely no intended use as "animation"?

---EDIT 11/2013---
The comments section on this and all posts related to the Burnt City Bowl are now closed, due to the inordinate amount of anonymous and slanderous comments left by people clearly bearing some type of political agenda (however construed). All comments made on this blog of such a nature will be deleted during moderation prior to being published.

Tuesday, February 17, 2009

Help my Science!

If you have roughly 15 minutes to spare, please take my comics related experiment! I am currently working on an a fairly extensive project that I've been able to put one portion of online. Hopefully this will be the first of many that will be accessible from my website, but right now I need your help!

Like I said, it should only take roughly 15-20 minutes, and participation will enter you into a drawing for a $50 gift certificate to Best Buy with a 1 in 50 chance of winning. This experiment will only be online for a limited time, so please help out soon!

--------EDITED 2/21--------

Data collection has now ended for this experiment. Thanks SO much to everyone who participated and/or linked to the experiment. This will be extremely helpful for this project. Stay tuned for more experiment postings on this site, or email me to be updated when more tests are available. Thanks again!

Sunday, February 08, 2009

The Pictorial and Linguistic Features of Comic Book Formulas

Neff, William Albert. 1977. The Pictorial and Linguistic Features of Comic Book Formulas. Doctoral Dissertation, University of Denver, Denver, CO.

Neff compares the patterns of formal properties for comics in different narrative genres of Adventure, Romance, Mystery, and Alien Beings or States. He analyzes the fields of panel shape (vertical, horizontal, square, circle), angle of view (lateral, high, low), and type of shot (close, wide) in comic panels. He also looks at pragmatic sentence types and parts of speech of stressed words from the text of comics.

His results show distinctions in the "formulas" of different genres, showing that different genres do use differing patterns for their distributions of these fields, and he then describes his interpretations for what those forumlas indicate about the genre (and vice versa).

While I don't doubt that such patterns for genres exist, this study had numerous problems. The categories for analysis were a little broad (only wide and close shots?) and often washed over in coding (diagonal panels were grouped as either horizontal or vertical). The interpretations of the genres' formulas also seemed a little like just so stories.

However, I take this entire study with a grain of salt because the sample size of his analysis is so small. For each genre, he uses only two comic books (pamphlets). While he does get statistically significant results using chi-squares, he pools frequencies across books, which eliminates any variation across books with no way to analyze it. Shouldn't he be using averages for this?

Two books per genre, and limited categories in the fields of analysis, are far too little to really get a sense of the patterns of an entire genre. His total number of panels in all was only about 530. In comparison, I consider my study comparing 300 panels in each of 12 Japanese and 12 American paperbacks (Cross-Cultural Space) to have been small in scope, and have just initiated a study of at least 200 books of varying genres and countries.

While I greatly appreciate the attempt at doing such corpus analyses (and am actively doing more), and especially like seeing it as a "hidden treasure" in the history of this type of study, this one unfortunately lacks the scope to be taken seriously.

Wednesday, February 04, 2009

Origins of narrative

A friend of mine forwarded me this facetious article about the narrative structure in films, which left me thinking about why exactly a consistent structure like the one that is found in Freytag's Triangle might be found so prevalently.

Studies on "story structure" have consistently found that people (in our culture) have better memory for stories that follow this arc than those that don't, indicating that it isn't just a vogue thing — it's cognitively advantageous. So, on the one hand, we might attribute this to an "innate" universal structure found in people's minds. This would be in contrast to the idea that there is an "archetype" floating out in the world that stories follow. Really, as with most all human behavior, there are no structures "out there" — the only place that such structures can exist is in people's minds.

Of course, the immediate comeback that might arise would be that not all cultures' stories follow this Aristotelian narrative arc. However, a simple fix around this would be that, like the patterns in languages, different cultures might externally have different narrative patterns, but the capacity of all human minds have structures that allow those diverse patterns to emerge.

For example, most all languages use Subjects, Objects, and Verbs, though they put them in differing orders. It may be the case that analogous categories exist for narrative, but that the Aristotelian narrative arc simply is one of the patterns that these categories are put into for a (i.e. our) particular culture.

Deeper though, we should think about the function of narrative in the first place. It seems implicit amongst most approaches that narrative is for telling stories. However, why might a mind have a need to tell "stories" specifically? Rather, I think stories are merely a symptom of the broader function, which would appear to be about ordering information — particularly about objects and events.

With such an organizational system in place, stories serve a cognitive purpose as a way to facilitate comprehension and memory. Entertainment and artistry for those stories is just a affective bonus.

Saturday, January 31, 2009

"The 99" Panel Discussion

As I mentioned in my last post, last week I had the pleasure of sitting on a panel discussion here at Tufts with alum Dr. Naif Al-Mutawa, creator and publisher of the comic The 99 and Dr. Chip Gidney of the Tufts Child Development Department. That video has now been put online and is viewable here (Windows Media, 1:09:09).

The description of the talk is here:

Naif al-Mutawa (A 94)
Creater of the new comic book series "The 99". "The 99" is a series of comic books based on superhero characters who battle injustice and fight evil, with each character personifying one of the 99 qualities that Muslims believe God embodies. Publisher Teshkeel Media is dedicated to "… cultivating and harvesting those themes intrinsic in our regional culture that will speak equally to children both in and outside of the Middle East." According to Forbes, the Teshkeel Media Group and "The 99" were one of the "top 20 trends sweeping the globe" in 2007. Dr. al-Mutawa attributes this to the "universal themes" in the series that transcend the Muslim backgrounds of its heroes. A principal author of the series, Dr. al-Mutawa is also the author of several children’s books on prejudice and race, and is a clinical psychologist and businessman by training.

Calvin "Chip" Gidney
Eliot-Pearson Department of Child Development

Neil Cohn
Department of Psychology

and moderated by
Julie Dobrow
Eliot-Pearson Department of Child Development, Communications and Media Studies Program

Co-sponsored by the Communications and Media Studies Program, the Eliot-Pearson Department of Child Development, and the Jonathan M. Tisch College of Citizenship and Public Service This program is made possible by a grant from the Tufts Diversity Fund.

Monday, January 26, 2009

I am Tufts University!

Apparently, someone over at the Tufts website thought my work was interesting enough that they decided to do a feature on me. SO... I am very flattered to report that, for this week only, I am featured on the homepage of the Tufts University website. From there you can link to an interview with me, also found directly here. The interview will be up indefinitely, but I think the homepage will only feature me for this week.

I also had the distinct pleasure this morning to sit on a panel here at Tufts with alum Dr. Naif Al-Mutawa, creator and publisher of the comic The 99, which is a globally distributed series with multicultural superheroes that each embody an attribute of Muslim values (though in the book this is apparently done without religiosity). The book is being praised for its multicultural and boundary-crossing qualities along with providing a positive alternative to many media representations of Islam.

We had a very fun and interesting panel discussion, which I've heard may appear online sometime soon.

Thursday, January 15, 2009

"Visual literacy"

The phrase "visual literacy" is one that is often bandied about these days, and has begun to grate on my nerves a bit — if only because it is a bit vague and vacuous in meaning. The phrase at this point is basically being used to mean a familiarity with anything that is an image and (usually) not text.

However, what kind of "literacy" is this exactly? The range of things covered by this term include vastly disparate material: diagrams, paintings, graphs, websites, comics, etc.

Not all of these items are processed in a similar way, and "literacy" for one does not necessarily equate with "literacy" for another. For example, I know people who are highly fluent in reading the visual language of comics, yet find "infographics" like flow charts mind-numbingly opaque (and vice-versa).

Such a phrase implies 1) that visual communication and expression is homogenous, and any diversity is washed over by its shared virtue of being "visual", and 2) that comprehension of one of these forms equates to or leads to equal understanding of the others.

This seems far from the case. The various things that are covered by this term have very different motivating structures and properties, and comprehension with one does not necessarily lead to the same skills with others (and especially does not imply that for production). Really, what we have is a number of disparate forms that each involve their own forms of fluency independently, despite a shared visual modality.

The implication that such diversity is homogenous is a kind of orientalism — likely just a view embodied from a culture entrenched in a verbal modality that is still grasping at a method of communication that it doesn't yet fully embrace or understand.

Tuesday, January 13, 2009

Anyone speak Croatian?

For anyone out there that speaks Croatian or just wants to check it out for the sake of novelty, my article "Un-Defining 'Comics': Separating the Cultural from the Structural in 'Comics'" (which was published in the IJOCA and the first chapter of my book Early Writings...") has just been translated in the Croatian magazine Zarez. You can find it scanned and posted here (Scroll down to the second article and you'll see my sciencey illustrations).

Check it out!

Sunday, January 11, 2009


Lots of stuff going on round here... First off, thanks to everyone for the birthday wishes yesterday, especially The Comics Reporter.

This week marks the first week of school, and I'm greatly looking forward to my first lecture on Wednesday of my Visual Linguistics of Comics course. It seems I'm once again a little bit of a rabble-rouser, since the university has never done a course accessible on podcasts like I've proposed.

So, the administration is still looking into things. If it does happen, it's likely the elearning version will start up a few weeks into the actual class. This isn't so bad though, since digital learning really doesn't have to be at the same pace anyhow, right? Stay tuned for more info, and email me if you want to be added to my list of interested e-students...

Wednesday, January 07, 2009

Elements of Page Layouts

Belgian graphic design student Chris Vosters sends along this 7-page pdf graphic essay that expands on my paper, Navigating Comics, on how people move through page layouts.

Chris does a great job of categorizing both explicit and implicit ways in which an author can direct the flow of movement across a page layout without relying on the rule system my paper describes.

I highly recommend checking it out!

Sunday, January 04, 2009

The interface of humor and narrative

This Zippy the Pinhead strip from a week or so cuts to the core of the canonical joke pattern in comic strips:

What's interesting to me about this is the choice of descriptors here. "Conflict" and "punchline" here have to do with joke telling, and could correspond to varying parts of an actual narrative arc.

For example, narrative often features a denouement at its end preceded by the Peak of actions. The Punchline could go into both Peak or denouement. In one case, the Punchline would be the apex of the actions, what the strip has led up to. In the case of a denouement, the Punchline would be a reaction to or resolution of that Peak.

So, what we actually have is two separate "schema" for narrative and for jokes:

Jokes: Intro-Set up–Conflict–Punchline

Narrative: Establisher-Initiation-Peak-Release

You could imagine these running parallel to each other, and then different parts hooking up into each other in varying ways. How exactly these schema interface depends on on the desired pacing of the joke I suppose.

Thursday, January 01, 2009

Resolutions and some such

Happy New Year! Bonne Année! 明けましておめでとう!

Various fun is brewing out here in visual language land. Given the several requests I've been receiving about my upcoming Visual Linguistics of Comics class, I'm now looking into finding a way for the course to be offered digitally via podcasts. If you're interested in taking such a course, please contact me. If I can show a list of names interested, the administration might be more likely to let me do it.

I'm also analyzing the data from a few very good looking experiments, so perhaps I'll be posting on those soon. I've got a few other interesting blog posts in the barrel coming up soon. Perhaps that will be my digital resolution: to blog more even when school is in session!

In the meantime, regarding psychology of resolutions as a whole, check out this great article by one of my department's profs.