Tuesday, December 21, 2010

Visual Linguist salute!

Journalista blogger Dirk Deppey has announced that today is his last day blogging on that site. Dirk has done a fantastic job as the "go to" aggregator of comic information on the web, and I'm very grateful for his promotion of this blog on his site. His help has been wonderful for promoting my research and recruiting for my online experiments. So thanks, Dirk, and whatever your future endeavors hold, good luck!

Thursday, December 16, 2010


Thanks to everyone who took my recent experiment, as well as to everyone that linked to it! I'm greatly appreciative for the support, and I assure you that this goes directly to insightful research on comics comprehension. This link actually had two separate experiments that it led to. I can't reveal anything about the second one, but I can describe what we were analyzing in the first one.

If you got the one that asked you to describe only single panels, we were analyzing how "active" or "passive" the descriptions of the actions in the panels were. These results will help interpret the results of another study that used these panels, as well as help design a future follow-up study.

Also, be sure to stay tuned to this site for future experiments. Several more should appear in the coming months.

Thanks again!

Tuesday, December 07, 2010

Take my experiment!

Once again I have a brief experiment online and I'd be extremely grateful for your help! This one involves looking at various strips and making predictions about them. Participation enters you into a drawing for a $50 Gift Certificate to Best Buy (either online or in store). Thanks in advance!

UPDATE 12/13: I have now collected enough responses, so thanks to all who have participated! If you didn't get a chance to participate but you'd like to take future experiments, please email me.

Monday, November 29, 2010

Review: Through Navajo Eyes

I recently discovered a gem of a book called Through Navajo Eyes by Sol Worth and John Adair. This is a fascinating book from the early 1970s where the authors taught various Navajo how to make films, and then observe the patterns and styles of their filmmaking. The book in its entirety can actually be read online here.

Five of the six Navajo that they taught had never seen a film, and were only taught film technology (but not about theory of editing). Despite not seeing films (though they knew what they were), the Navajo were found to be amazingly adept filmmakers, particularly in the editing process, and had a remarkable knack for visual memory of shots.

There are many parts of this book that are interesting, particularly in the structure of the films the Navajo made. The authors describe that their films often did not employ a standard narrative progression with a building of tension. Rather, they began with the event being completed, and then the rest of the film worked to get show how they got to that point.

They also do not seem to care about continuity editing to blend shots of actions together into a seeming continuous stream. Jump cuts abounded and were not viewed as unusual. Their films featured a lot of walking (for the sake of walking), and it seemed that the emphasis on motion was more important than the continuity of actions. Thus, if a person was walking in a shot, then suddenly jumped in another shot to a place further down the road, this was not viewed as unusual.

This cutting without continuity appears different than the films made by inner city American teenagers, which the authors also had studied. Continuity editing is described as being implicitly learned by these students, who did it without being taught the theory (but had seen Hollywood movies that use this technique).

With the pervasiveness of movies in today's world, I'm not sure if doing a similar project in these times would work quite as well. However, the ubiquity of digital film editing software would certainly make it easier for individuals to make films.

Overall, the book is a fascinating study of film and visual narrative, motivated by an interesting premise.

Worth, Sol, and John Adair. 1972. Through Navajo Eyes: An Exploration in Film Communication and Anthropology. Bloomington: Indiana University Press.

Tuesday, November 09, 2010

Going off the grid for a bit...

I've been a bit absent from blogging lately, largely due to being crazy busy with work, but next week I'm legitimately going off the grid for a martial art retreat. So, I figured I'd at least give an update about things.

First off, my manuscript about my model for the structure underlying sequential images has grown to be quite massive. I'm now up to 197 pages (single spaced, 1" margins), and it's still growing! The plan is to finish off this paper by Winter break, and then figure out how best to get it out and read by others.

I've also recently started the planning and preparation of what I hope to be my dissertation projects about sequential image comprehension. I have two fantastic assistants working under me who are helping prepare the stimuli, so hopefully a new experiment will be online soon to help get things underway.

Added to that, I'm currently preparing a few articles to submit for publication. Looks to be busy times ahead...

Thursday, November 04, 2010

Review: "Copying and Artistic Behaviors"

Smith argues that the negative views on "copying" demonstrated by art educators since the 50s is misplaced in some contexts. She claims that some forms of copying are good, and the relative value of copying is based on three factors: need, model, and process. She examines varying fields through use of a corpus of comics produced by American children, noting that themes and genres are copied greatly. She didn't find that the children copied the drawing style as much.

My curiosity is whether this is due to lack of practice/exposure though. The examples given by a child with "unusual ability" seem hardly on par with Japanese drawings of children of the same age that copy manga en masse. This child did copy various elements of drawings, though not absolutely. For instance, when copying Charlie Brown, he imitated parts but altered/left out others. Another child drew the typical "lumpy" figure of Captain America to show his musculature. Smith conjectures that his intent was to draw someone "strong" as opposed to drawing a bicep in particular.

To this extant, these children's copying seems to be drawing characters/features to the point of recognition — not iconic match. In other words, they're trying to convey concepts visually, not create "realistic" pictures (or even "accurately" imitated images).

While interesting to see much support given to imitation, most of it is not structural, and still maintains an "Art" perspective. The "need" assigned to copying is largely social or emotional/psychological, not structural or cognitive. (For instance, it says imitation suits a child's need to "play out" conflict in fantasy, as opposed to saying that children copy because their brains are pattern seeking machines).

Social need is Language-like though, as it heralds conventionality. She also marks copying as important as a natural behavior in socialization, since "younger children initiate copying as a means of acquiring desired knowledge" while "older children want to master images representative of their culture" (147).

Also interesting was her statement why she wanted to look at comics in the first place: "Comic strips are of interest because children frequently and spontaneously initiate copying of them despite disapproval" (148). No citation is given to this statement, but are comics copied more than other forms of visual communcations in culture? (it wouldn't surprise me if the answer is "yes") And, if so, doesn't that say something about the structure of the stimuli in relation to the human mind — like maybe these signs are somehow attuned to acquisition and socialization?

Smith, N. (1985). Copying and Artistic Behaviors: Children and Comic Strips Studies in Art Education, 26 (3) DOI: 10.2307/1320320

[Originally posted 5/15/07]

Thursday, October 14, 2010

6 Q & As

Webcomicker William George posts a quick Q&A with me on his blog. This is part of a series of quick interviews he's doing with various people connected to webcomics. I guess I qualify enough because of my older webcomics and writings for comixpedia (now comixtalk), but I'll take it. Check it out!

Tuesday, October 12, 2010

Kid's sequential drawings

This is a summary/review of an article I thought had particularly compelling evidence for why understanding sequential images is a learned trait. Highlights are all mine...

Narratives of urban Japanese children (manga) were compared to those of village Egyptian children. The argument was made that development differs based on graphically “rich” versus graphically “poor” environments. Egyptian children teach each other how to draw (“world of childhood graphic imagery rather than adult imagery” p. 10). Egyptian children’s drawings were floating, static, 2D, and symmetrical– more a reflection of perception (“intrinsic and intuitive bias towards simplicity”…”reflect humans’ innate preference for simple nonoverlapping shapes” p. 15). Japanese children’s drawings were often occluded, cropped, with lots of visual elements, had some sort of plane to ground the images, and employed “cinematic” techniques– similar to that found in manga (Japanese children by 8 or 9 may have passed the point where they are inspired by innate factors– p.15).

Over 2/3 Egyptian children drew narratives where the contents of one frame was not sequentially related to the next frame. All Japanese children drew sequential narratives – and at a “higher level of story structure” (p.16). “Japanese children were three or four times more likely to depict a related series of events or process.”

Their conclusion is that the urban versus village lifestyle, plus other cultural factors encouraging drawing are what lead to the difference in representational ability. My response would be that its not the urban/village lifestyles that cause this, but exposure to VL and practice with it. Japanese kids live in rich visual language culture (manga), and actively develop those this graphic fluency. They do note though, that Egyptian children did not have access to comics, and “television for the Egyptian children seems not to provide a functional model for producing the structure of graphic narrative plots” (p16). Manga, of course, does provide that for Japanese children.

This is another example of how looking at graphic creation through a Language perspective alters the way data is interpreted. Because drawings look like what they represent, the Art POV will attribute influence to all sorts of perceptual and societal influences. A Language perspective focuses mainly on the exposure and devlopment of those particular structures in their cultural surroundings: if you're going to produce (visual) language, what (visual) language is around you?

ResearchBlogging.orgWilson, Brent, & Wilson, Marjorie (1987). Pictorial Composition and Narrative Structure: Themes and the Creation of Meaning in the Drawings of Egyptian and Japanese Children, Visual Arts Research, 13 (2)

[Originally Posted 1/6/06]

Saturday, October 02, 2010

New York ComicCon and a video

Though I hadn't planned on it originally, it looks like I'll be attending the New York Comic Con this year, randomly enough as a guest of Archie Comics. After ComicCon in San Diego, they have become interested in my research, especially related to education, so they've invited me to come. I'm only planning to go on Friday, and my plans are pretty loose right now during the convention (not speaking), so if you have any desire to meet with me, please let me know!

In the meantime, here's a video of my friend and true genius of experimental comics, Jason Shiga, explaining in a video how to create interactive comics (something he's a master of):

Monday, September 27, 2010

Predispositions for drawing

A recent comment to another blog post raised some interesting issues so I figured I should bump it up to a full discussion here.

The basic issue is whether drawing in a realistic style (proper anatomy, shading, depth, etc) is somehow antithetical to our cognitive predispositions for drawing. I've argued before that drawing involves the deployment of graphic patterns in our minds in various ways, and that fluency is a proficiency in this system of representation.

There seem to be two issues here regarding the system of drawing:

1) What can we do?
2) What are we predisposed to do?

The answer to the first question is that we absolutely can draw realistically. Indeed, we can even develop mental patterns that can generate these realistic images. This is what creates a "style" of drawing. No one would say that Jim Lee or many other modern comic artists are "cartoony", yet they certainly use consistent patterns in their drawings that can be tied directly to their "styles."

However, I would say regarding question #2 that we are not predisposed to draw realistically. The evidence I've seen seems to suggest that our natural system of drawing is based more on representing contours and basic patterns. The more realistic aspects of a drawing system (perspective, depth, shading) are always the things that are more struggled with and require explicit teaching to learn (i.e. they won't be acquired effortlessly without some form of external instruction).

My advisor has recently been addressing a similar issue in language: what is it that a language speaker naturally is predisposed to know without any exposure to an external system, and what is learned (and relatedly, what aspects of cognition do we share with other primates and what has evolved to uniquely enable humans to speak). His argument, which will hopefully be making an appearance in a journal sometime soon, is that there is a certain set of innate principles that humans have regarding speaking, and that modern language has added structure on top of these deeper predispositions.

This is largely the way I view drawing. Our cognitive predisposition is for certain types of graphic representation: line drawings built of patterns that lack perspective and depth, though can use occlusion, etc.**

On top of this is the potential for explicit instruction for further iconicity: "accurate" anatomy, shading, perspective, etc. This is why much pre-Renaissance art lacked these iconic features. Because we aren't predisposed for it, "point perspective" and "shading" was a "discovery" and not something learners mature into naturally.

This doesn't mean we "can't" or "shouldn't" do it, but it does mean we aren't wired to do it without instruction.

** For an excellent work on what the developmental trajectory of drawings looks like without influence from learning an external system, check out John Willat's Making Sense of Children's Drawings.

Wednesday, September 15, 2010

Gestures in comics

A doubleshot of reviews**:

ResearchBlogging.orgFein, Ofer, & Kasher, Asa (1996). How to do things with words and gestures in comics Journal of Pragmatics, 26 (6), 793-808 DOI: 10.1016/S0378-2166(96)00023-9

This study looked at the role of gestures in comics (specifically, those in the European comic Asterix). The study had people interpret the meanings of both panels from the comics, and of photos where people took on similar poses. The backgrounds of the panels were erased, so there was no context for the gestures. In one part, they were asked to write possible dialogue for the gestures, and in another task they were given potential meanings and asked to assign them.

It concludes that gestures in comics are interpreted the same as ‘real life’ gestures, and that the meaning imbued in them comes from the ingesticular force (i.e. the intent of the expression) rather than the propositional content of the accompanying speech (in word balloons). One interesting tidbit noted that some people said the photos were actually harder to interpret than the comics panels (though the stats disputed this). If this were true, then it would support McCloud's insinuation that cartoony images are more "base" than realistic ones. I'd like to know the degree of fluency the subjects had with reading comics and whether people with more "comics" experience rated higher or lower in this regard.

Raecke Jochen. 1999. Using Comics as Data for Research into the Connection between Pointing Gestures and Deictics. In E. André, M. Poesio, and H. Rieser (eds). Proceedings of the Workshop on Deixis, Demonstration, and Deictic Belief at ESSLLI XI.

This is a short and hard to find article that I had to scour several libraries to find. This study uses comics to analyze the relationship between deictics (words that "point" to something else, like pronouns) and gestures in Serbo-Croation. His method codes a corpus of comics comparing the relations of the images' gestures to the conent of the speech balloons. He finds that pointing gestures by far dominate the gestures, and pointing gestures alone do not fulfill the meaning of the representations (i.e. multimodality is necessary). This isn't surprising, since pointing gestures are indexical, which means that they only indicate meaing in something else (the same way a pronoun refers to a different element for meaning).

**This post was originally posted 12/23/05

Tuesday, September 07, 2010

Review: Brain damage and ordering of panels in comic strips

I recently reviewed an older study of brain damaged individual's comprehension of final-panel jokes in comic strips. Here's another paper that explores brain damage and the ordering of panels in sequences.

Participants were asked to arrange scrambled parts of a story into their accurate order, and the authors compared the abilities of numerous types of brain damaged patients. Participants reconstructed both six-panel comic strips as well as verbally translated versions of those strips (in two forms: descriptions with complex grammar and those with simple grammar).

Individuals with global damage to their left hemisphere and Wernicke’s aphasics (also a damage to the left hemisphere) did the poorest on reconstruction of both types.

Patients with right hemisphere damage did poorly in reconstructing comic strips, but not verbal stories. Broca’s aphasics (who have damage to the frontal left hemisphere) showed the opposite trend: poor reconstruction of verbal stories but decent performance on comic strips.

Broca's aphasics are largely recognized as having deficits with issues of hierarchic ordering, particularly grammar in language. So, when they speak, they may be able to create meanings, but they struggle to produce combinations of words in sentences. Wenicke's aphasics are known for the opposite: their grammar may be intact, but they have problems with meaning. This means they can speak in full sentences, but those sentences will make no sense.

These findings imply that both hemispheres are involved in the comprehension of verbal and visual narratives, but do so in differing ways. The right hemisphere appears to guide picture reconstruction more than the verbal comprehension, which appears more effected by left hemisphere damage.

However, one flaw in the study is that it is a little unclear on just what traits the participants are manipulating. In other words, what aspect of comprehension is being affected by their brain damage: Narrative? Semantics/Meaning? These are not the same thing, and it's hard to tell with the task what is being targeted. It is also hard to tell which of these is being damaged by the brain damage — perhaps in some cases the brain damage affects narrative, but in others semantics.

Thus, while it is nice to have dissociable findings and clues to comprehension, the design and theory underlying the experiment make any solid conclusions hard to discern.

ResearchBlogging.orgHuber W, & Gleber J (1982). Linguistic and nonlinguistic processing of narratives in aphasia. Brain and language, 16 (1), 1-18 PMID: 7104674

Friday, September 03, 2010

Back to school...

School is starting up next week, which means I suddenly have lots more stuff appearing on my plate. I'm still plugging away at my theoretical paper (now at 140 pages and growing!) as well as preparing a few other papers for submission to journals. Once I finish that paper, I'll be jumping back into the grind with more experiments. I'm also excitedly taking my last required course of grad school this semester, which is on the nature of scientific discovery. Should be fun!

ResearchBlogging.org Otherwise, I have yet to project what sort of timing I'll have for blogging coming up. However, I recently joined the site Research Blogging which is an aggregator for blog posts about peer-reviewed research. So, I may start reposting several of the article reviews that have been done on the blog over the years so they'll also be picked up there (and reappear for any new readers who may have missed them the first time around).

Wednesday, August 25, 2010

New Essay: Extra! Extra! Semantics in Comics

I'm excited to say that I've just had another article published. This one is entitled "Extra! Extra! Semantics in Comics!: The conceptual structure of Chicago Tribune advertisements" and appears in the latest edition of the Journal of Pragmatics. Here's the abstract:
Recently, increasing attention is turning to comics as a graphic domain using similar cognitive processes to linguistic forms. As in the verbal and manual modalities of expression, various semantic structures arise across sequences of images in interesting and effective ways. This piece examines metonymy, conceptual metaphors, and blending across a three-panel pattern used in strips from an advertising campaign by the Chicago Tribune newspaper.

I originally had the idea for this paper when I was living in Chicago. I kept seeing these comic strips as advertisements all of the city and they really struck me as interesting since they were using principles I knew from linguistics. As it turned out, I had a class with professor Christopher Johnson (a former advisor of mine) where those ideas were being studied, so I brought in the strips for class discussion and wrote the paper as a final essay.

I think many of my papers end up plotting out new ground for research, or developing some new theoretical tools. This is a rare case where I simply saw interesting phenomena out there and applied existing ideas to illustrate it. That made it fairly fun though.

This paper was originally posted on this site for several years under a different title. This version has been overhauled and updated, so I recommend it for any who are interested in meaning-making in sequential images. Enjoy!

Sunday, August 22, 2010

Reading comics for work

I admit it, I have a pretty sweet job. Among the paper writing, class teaching, and data gathering (all fun), it is true that at least a portion of my time is devoted to reading comics. A lot of comics. I end up combing through hundreds of pages of comics just to find examples that might be worth discussing in my papers.

For example, this summer I've been working on a massive paper detailing the inner workings of how people comprehend sequential images. And by "massive", I mean that I still have numerous sections left to fill in, and the paper is already 130 pages.

I spent last weekend combing through a giant stack of Calvin and Hobbes looking for interesting strips. Unsurprisingly, there were many. (It's a rough life...)

One of the things I've noticed recently is that very few comic sequences really faze me anymore. I seem to be able to account for most everything I come across in my existing theories, which is both cool ("yay, my theories seem to be working!") and troubling ("I need to find more strips with things I haven't thought of!").

So, if you happen to have any sequences that you think are a little strange or challenging (especially strips), please send them my way! Maybe I'll even post them here and give them some analysis...

Monday, August 16, 2010

Lightbulbs over the head actually do give insight!

A friend of mine and fellow grad student in the Tufts Psychology Department (Mike Slepian) recently had a paper come out that's been getting some press lately. Interestingly enough, it relates to the common cartoony emblem of the lightbulb above the head. This symbol is a conventional sign for "inspiration" in comics and cartoons, so the authors wanted to see if real lightbulbs could actually give people inspiration.

Lo and behold — they did! In a series of experiments, participants were asked to do logic problems and near them light was turned on from either a lightbulb or an ambient overhead light. Participants with a lightbulb consistently completed more logic problems correctly than those with the overhead light.

This indicates that, not only do the conventional signs of a lightbulb represent inspiration, but that meaning has become entrenched enough that it actually feeds back on performance in cognitive tasks. That is, the graphic symbolism has affected the way we think and behave.

The idea that language effects thought and behavior is called "linguistic relativity" or Whorfianism and is a highly contentious debate. The vocabulary of visual language may have some degree of linguistic relativity as well — at least for this particular sign. I'd love to see more research like this with other graphic signs. I'm not sure what my prediction would be, but it'd certainly be interesting...

Slepian, M.L., Weisbuch, M., Rutchick, A.M., Newman, L.S., & Ambady, N. (2010). Shedding light on insight: Priming bright ideas. Journal of Experimental Social Psychology, 46, 696-700.

Monday, August 09, 2010

New Book: Meaning and the Lexicon

It slipped my mind to post this a few months ago, but my advisor Ray Jackendoff has just a had a new book come out that is worth picking up entitled Meaning and the Lexicon. It features a collection of his papers tracing the development of his theories over the past 35 years. Pertinent to me, not only does this work frame the work that I am doing, but the book features new illustrations by yours truly!

There are several great and interesting chapters in this book, particularly one on the relationship of the conceptual system and language to the visual system, addressing the question of "how do we talk about what we see?" This is of course interesting to me because the bulk of the things I talk about are visual.

There's also a classic article about "Parts and Boundaries" which concerns the underlying principles guiding concepts having to do with whether things have internal parts (are they divisible into pieces and can retain the same idea? Ex. "water" broken apart is still water, but a "dog" is not...) and do they have boundaries (ex. a "lake" has a fixed boundary, but "fog" does not).

This latter article was very influential on me long before I even came to Tufts to work under Ray, and it inspired my approach to explaining the underlying semantics behind word balloons and thought bubbles (formerly online in the essay "Interfaces and Interactions", now taken down, retooled and hopefully appearing in a journal soon). Amusingly, I didn't know it when I was drawing the pictures, but it turned out that I drew illustrations for this refurbished chapter in the book. I've come full circle!

Monday, August 02, 2010

Picasso's Ghost

I've got some other posts in the works, but in the meantime an old friend of my father's made this fun video about Picasso, Cezanne, and the inspirations of great art. Enjoy!

Sunday, July 25, 2010

Comic-Con 2010, PhD Comics Cameo, Facebook

Today closes another wild and fun ComicCon. My talk seemed to go fairly well (thanks to those who came!), and I greatly enjoyed the discussions with people afterwards. This was the first year I got to present actually experimental brain data on comics, which I'd been looking forward to for awhile. I only wish I had more time to answer questions, so if you didn't get an opportunity to ask what you wanted to, email it to me and I'll discuss it on the blog!

Anyhow, I had a particularly great time this year, so thanks to everyone who I saw and interacted with!

Amusingly enough, I also make a cameo this weekend in the latest strip of PhD Comics by Jorge Cham. We met at last year's convention, and he's included me in his summary of the Con. I had a great time with him, made even more fun by the fact that I read his strip nearly every day already (and you should too!).

Also, I now have an official public Facebook page. I'll be using this page as another place to find updates on papers, lectures, and notices on opportunities to participate in research. So, if you enjoy my work and or would like to follow it via updates on Facebook, please add me!

Wednesday, July 21, 2010

ComicCon 2010 Reminder

Just a reminder that my talk at ComicCon, "This is your brain on comics", will be on Friday from 1-2pm in room 26AB. I'll be speaking about my recent experiments looking at how the brain processes the sequential images in comics. Hope to see you there!

Monday, July 12, 2010

Soo Bahk Do Presequenced sparring, 2010 Nationals

This isn't "visual linguistics" related, but this last weekend I competed at the 2010 Soo Bahk Do National Festival. In the advanced ranks division my partner, Master PJ Steyer, and I won first place with this coreographed routine, so I thought I should share.

I'm the one who starts on the right with the shaved head. Supreme kudos to Noelia Lago for the great filming!

Sunday, July 11, 2010

Comicon 2010: This is your brain on comics

The official ComicCon schedule was just released, and it looks like my talk will be Friday 1-2pm, room 26AB. Here's the official description of the panel:
1:00-2:00 Comics Arts Conference Session #7: This is Your Brain on Comics Theory— Julia Round (Bournemouth University) argues that From Hell relies on a comics aesthetic, in which all panels are co-present in the page's spatial layout, to represent time as co-present and nonlinear. Neil Cohn (Tufts University) discusses two psychology experiments examining what happens in the brain when people read comics. Room 26AB

I'll post a more in depth description of my own talk soon, but yes, when it says "brain" it really means it: I'll be showing data from my brainwave experiment using comics.

Monday, June 21, 2010

Brain damage and comprehension of comics

Bihrle, Amy M., Hiram H. Brownell, John A. Powelson, and Howard Gardner. 1986. Comprehension of humorous and nonhumorous materials by left and right brain-damaged patients. Brain and Cognition 5:399-411.

This study compared the narrative comprehension of people with damage to either the right or left hemispheres (RH vs. LH) of their brains (often from strokes or physical damage, like concussions).

They used a "completion task" where they were shown three panels from a comic strip, and were asked to provide either a humorous ending or a non-humorous ending with a choice of panels. The humorous ending was the original strip's final panel. The non-humorous ending was manipulated to have one of four varying degrees of coherency related with the strip:

1) ordinary endings with non-funny events that still made sense
2) endings associated to the strip but non-sequitur, such as featuring water if the strip had water in it
3) totally non-sequiturs that had no relation to the strip and weren't funny
4) humorous endings but that were non-sequitur given the context of the strip

Overall, they found that people with right hemisphere damage made far more errors than Left hemisphere patients. Left hemisphere damaged patients did far better, and several actually hit the ceiling of good performance.

In errors for the task that asked them to provide humorous endings, Right hemisphere patients generally erred by choosing endings that were surprising but not coherent, while left hemisphere patients chose unsurprising but coherent endings. Right hemisphere damaged patients were most drawn to non-sequitur endings of various types, and not to the ordinary ones. Left hemispheric damaged patients showed the opposite results.

However, when asked to provide a non-humorous ending, both types of patients generally chose endings meaningfully associated to the strip but still non-sequitur.

These results are consistent with the poor understanding of verbal narrative and jokes by right hemisphere damaged patients. Overall, they do seem to be sensitive to the formal properties of jokes (such as that an ending should be surprising), but they seem unable to establish coherency between panels.

Monday, June 07, 2010

2D Drawing: comprehending and producing

Scott points to this interesting article about the cognition motivating line-drawings. It makes the claim that people with lazy eyes may have a better ability to draw 2D figures because they have trouble with depth perception to begin with.

Mostly I find this prospect intriguing, though I do somewhat question the assumptions motivating this idea, revealed in this quote:
"Although we experience the world as three-dimensional (thanks to the separation of our two eyes, which produce two different vantage point, and the visual cortex, which reassembles the images into a cohesive landscape), recreating that world in art and film has been challenging."

The implication here is that the capacity to draw is "re-presenting" the perceptual (i.e. 3D) world in 2D form. Rather, I believe the function of drawing is simply to express our concepts in the visual-graphic domain — a modality that is most suited to iconic representations (i.e. "iconic" in the Peircean sense that means "meaning through resemblance"). It's not that we're trying to recreate what we see, but that we're trying to express what's in our heads graphically.

To this end, I think it is important to address a rather large part of the story: the ability to draw in the first place.

More than just the reconversion of seeing into drawings, it's important to consider how we are able to produce line-drawings. One would assume that these capacities would be related — that our perceptual system is adapted to view line-drawings just as much as our minds are built to produce them (I make no claim on which would have evolved first).

I have trouble pinpointing a reason that having impaired depth perception would hinder or advance the ability to draw in circumstances where you're not doing "life drawing." Drawing "from memory" isn't necessarily the pulling up of mental images in place of not having an actual thing to look at. Evidence seems to indicate that drawing pulls from pre-established graphic patterns stored in our minds that are deployed in different ways. When we draw from perception, we route our vision through the schematic information we use to draw. This is why people's life drawing reflects their own "style" — they are those mental patterns. In some ways, learning to draw proficiently with "realistic accuracy" may be the suppression of these schemas.

Further, while the article mentions that monkeys can perceptually recognize what objects are in line-drawings, they cannot produce them. However, babies take minimal amount of time before they start producing them, and without any sort of explicit teaching.

This is a dissociation that seems relevant: if what separates human babies and monkeys is the capacity to produce line -drawings but not to perceive them, it seems like a particularly important part of the story to address in terms of cognition and evolution.


One additional note: anecdotal evidence about pictures of artists with lazy eyes seems like suspect evidence to ground a theory on (and somewhat of population bias). It also mentions Babe Ruth having a lazy eye... does that mean depth perception is a drawback to hitting baseballs? I'd think that'd be very important there!

Monday, May 31, 2010

8 years!

Today, May 31st marks 8 years of emaki.net! That's quite the long time for me... I had just graduated college and the Lakers were in the playoffs back then. Funny enough, I'm now still in school (ahem... grad school) and the Lakers are once again going to the NBA Finals. I guess the more things change...

Anyhow, this summer looks to be fairly eventful. I have quite a few blog posts lined up for the coming weeks, so hopefully we'll be seeing some more constancy on here. And, in July I'll be giving my annual ComicCon presentation, this year discussing my Masters project that analyzed the comprehension of sequential images in the brain. Just to whet your appetite, the title of the talk will be "This is your brain on comics."

Monday, May 24, 2010

Panels in Japanese vs. American Mainstream/Indie comics

This semester my student Amaro Taylor-Weiner did a great senior project following up to my cross cultural study comparing Japanese and American comics. He was one of my students in my "Visual Linguistics of Comics" class last Spring, and he jumped at the chance to do some work of his own.

We compared the amount of information held in 300 panels from each of 10 Japanese manga with 20 American comics of different genres: 10 Mainstream books and 10 "Indie" comics**. We found that the genre differences in American comics were marginal compared to Japanese manga, which used types of panels in very different ways. A poster of the study can be read here.

Interestingly, not shown in the poster (outside the error bars on the graphs), the patterns found in the Japanese panels were far more consistent across books than panels in American books, which were much more variable. The most variable were actually the Indie comics — some seemed to pattern like the Mainstream books and some like Japanese manga.

We interpreted these results as having connections with aspects of Asian versus American cognition related to research on attention, and further support that there are indeed different cultural visual languages.

He did a great job on the project, and I wish him the best as he goes on to his post-graduation endeavors!

**Thank you to all the great comic companies that donated books to these efforts! I'm still accepting donations from anyone who wishes to contribute (especially international books!!).

Thursday, May 13, 2010

Advice for aspiring comic theorists

I read just about any paper about theories of comics I can get my hands on — some are pretty good, but some leave a lot to be desired. This should maybe be expected from a still-burgeoning field. Usually, when aspects of a paper have troubles, there are some consistent problems. So, here's some advice for potential "comic theorists":

1) Don't cast your theories so wide they are difficult to validate. Broad sweeping claims should only be made if they can be backed up by examples and/or citations. It's better to be specific and explicit in your theories.

2) If you're going to create a theory about comics, base it on the properties found by analyzing actual comics instead of pure theorizing. Often times people get so involved dreaming up of logical possibilities that could occur (or borrowing them from other theories) that they don't notice what does occur. At the very least...

3) Test your theory on actual comics (lots of them!). If you don't find your theory accounts for things you find going on in them, modify your theory appropriately.

4) Develop your theory with conventional examples, not exceptions. Often times the exceptions make for the most interesting examples, because they are noticeably different than "normal" usage. Granted, you can use those exceptions as clues to how conventional aspects of the medium work. However, if your theory is just about exceptions, it won't generalize to "normal" aspects of the medium too.

Thursday, April 29, 2010

Book Chapter: Manga

It isn't entirely a new paper, but my paper "Japanese Visual Language" is now published in the anthology Manga: An Anthology of Global and Cultural Perspectives now out in paperback. A brief description of my chapter:

Over the past two decades, manga has exploded in readership beyond Japan, and its style has captured the interest of young artists all over. But, what exactly are the properties of this "style" beyond the surface of big-eyes and "backward" reading? This paper explores the structural properties of the visual language underlying the "manga style," how it works, and how it differs from the visual languages in comics from other parts of the world.

The book on the whole though seems to offer a fairly wide range of interesting papers that go quite in depth, and is well worth picking up for anyone interested in study manga.

Friday, April 23, 2010

New essay: The limits of time and transitions

On the *NEW* publication front: The first issue of the new journal Studies in Comics went online today, and I have a brand spankin' new article in it. The first issue is available to read for free online. Here's an abstract of my paper:

The limits of time and transitions: challenges to theories of sequential image comprehension

The juxtaposition of two images often produces the illusory sense of time passing, as found in the visual language used in modern comic books, which creates the sense that this linear sequence presents a succession of moments or temporal units. Author and theorist Scott McCloud took this view to an extreme, proposing that sequential images are guided by a notion that ‘time = space (McCloud 2000), because this temporal passage occurs on a spatial surface. To McCloud, this ‘temporal mapping’ results in a movement of time with a movement of space. This sense of temporality then is the ‘essence’ of comics, which is manifested in McCloud’s taxonomy of transitions of panel-to-panel relationships (McCloud 1993). While less specific, this same type of ‘essence’ of connection can be reflected in Groensteen’s types of ‘arthrology’ across a linear sequence or disparate panels in a broader text (Groensteen 1999).

However, numerous problems arise with McCloud and Groensteen’s approaches to graphic narrative. This article will explore how the linearity of reading panels and the iconicity of images create various false assumptions about the conveyance of meaning across sequential images’ depictions of space and time. With numerous examples, it will argue that any linear panel-to-panel analysis (such as McCloud’s (1993) panel transitions) or loosely defined principles of connection (such as Groensteen’s (1999) ‘arthrology’) between sequential images are inadequate to account for their understanding. The conclusion is that sequential image comprehension must be thought of as the union of conceptual information that is grouped via unconscious hierarchic structures in the mind. As such, the study of the comprehension of the visual language used in comics must be placed in the cognitive sciences.

Saturday, April 17, 2010

New blog address!... roughly same as the old...

Hey, my blog has now just slightly moved addresses over to: http://blog.emaki.net/. This was one of the previous address, but now it's the main address along with http://www.thevisuallinguist.com. You will be automatically redirected in 30 seconds, or you may click here

For feed subscribers, please update your feed subscriptions to:

Wednesday, April 14, 2010

Metaphors go boom!

Derek Kirk Kim uses a fun and common conceptual metaphor in today's installment of his ongoing series "Praxis & Allies" (which you should be reading). Here, the character's head explodes on discovering his crush actually likes him.

"Conceptual metaphor" is a notion that's been prevalent in some circles of linguistics for the last 30 years, and is a phenomena where one domain of ideas is mapped onto another. The metaphor that Derek uses is "Emotion is Hot Fluid in a Container" — the emotion constitutes "pressure" in the head (the container) that then can "erupt" when it "overflows." Emotion isn't actually hot fluid in a container, but we map the idea of emotions onto the domain of a container.

There have actually been several papers written on this topic with regard to comics, usually describing Anger. The common visual sign of smoke coming out of an angry character's ears directly links into this "Anger is Hot Fluid in a Container" metaphor. But, here Derek uses it just for an emotional overload. A similar usage was done years ago in Journal Comic by Drew Weing.

As has been argued by many, using these sort of cross-domain metaphors is a great way for the graphic form to visually portray things (like emotions) that aren't otherwise visible.

Monday, April 05, 2010


I'm very excited to say that, after working on this project for 2.5 years, I'll finally be defending my Master's project — "Balancing Grammar and Semantics in 'Comics': Global Structure in Sequential Image Processing" this coming Monday April 12th here at Tufts. The presentation will describe two experiments that together show converging evidence that the comprehension of sequential images — as in comics — uses a grammar, similar to the way that sequential words use a grammar. Here is my abstract in Haiku:

Image sequences
Grammar, Meaning — separate?
RTs, ERPs.

It is a public defense, so if you actually want to come, you're more than welcome to email me for more info. A shortened version of this presentation will be what my talk is about at Comic-Con this year...

Friday, March 19, 2010


So, I unfortunately haven't had much an opportunity to blog lately. My apologies to the usual readers! Some updates...

I have been working hard at finishing my latest projects, which I had the fun opportunity to present as a poster at the CUNY sentence processing conference out here in New York City. To any readers that found this blog through the conference, thanks for checking out my poster and the site!

I'm currently working on writing up those experiments, which I'm very excited about. I also have a few publications in the works, which I'll be updating about once they're due to make an appearance. Plus, with my assistants, we've have been working on some projects that should have some exciting results soon.

I do have some backlogs of intended blog topics, so hopefully I'll be back to posting more entries soon...

Monday, January 25, 2010

Action Stars and Smoke-veiled fights

I've posted every now and again about a convention in comics that I've called "action stars", where a whole panel is replaced by a star shaped "flash" that essentially represents "event happens here!" but doesn't show that event. I've likened this to being like a pronoun in the visual grammar, since it can replace the Peak events of the sequence, just like a pronoun can replace a noun (or noun phrase).

Over the past year I've run some successful experiments using action stars, and am planning a few more of them. But, I've also had the lingering question whether there are any more of these "visual pronouns" out there...

And I think I've found one.

Another common piece of visual morphology is the "smoke-veiled fight" (alternative names welcomed), where a big puff of smoke is shown with arms and legs sticking out, to stand for a fight occurring, which can also take up a whole panel:

Some interesting contrasts can be made between the smoke-veiled fight (SVF) and action stars. First off, SVF panels are much more restricted; they can only appear for fights, whereas action stars can go on almost any Peak panel. We might write this out this difference in meaning formally as:

Action Stars: [Event: X]
SVF: [Event: FIGHT(A,B,...n)]

This basically says that an action star carries the unspecified meaning that an Event "X" occurs, but SVF panels show an Event of "Fighting", consisting of at least characters A and B up to "n".

Also of interest is that while both depict "events", the nature of those events is intrinsically different. Action stars show a single event, while SVF panels show a duration. Notice, you can't glean the sense of duration from an action star, nor can you interpret the SVF as a single event. But, the difference is there — even though in neither one can you actually see what events are "actually" happening!

Tuesday, January 19, 2010

"Defining Comics" video

Patric Lewandowski offers this video lecture of his attempt to define comics, based on his earlier column from Comixtalk. He covers a lot of ground, meshing numerous memes of comics theory.

Ultimately, I do disagree with most of his points, for reasons I've described elsewhere**, but it's at least interesting to see him present it all together, and I do like that other people are at least trying to address these issues.

While I've defended people's attempts to define comics before, from the visual language perspective, the whole issue of "defining comics" does seems a little strange, and likely stems from McCloud's big thrust to do so in Understanding Comics.

McCloud's guiding rhetoric was a division of form and content. For him, the form was comics and the content was the genres that appear in comics. But, you can take this one further, since comics are only a "form" if you presume them to be.

Really, comics are made of two mediums: text and sequential images. These can be the "form" and the notion of "comics" is the content. Truly, as Patric tries to get at, text and/or images appear in lots of places, but only sometimes are they called "comics." This says to me that "comics" are not a thing definable by those elements at all as a type of "form."

Rather, comics are written in text and images the same way that novels or magazines are written in text. From this perspective, debates over "what are comics?" are rendered similarly to "what are novels?" or "what are magazines?"

**Though I will add emphatically that my theory of visual language as he presents it there is fairly misleading. Visual language is NOT just about iconography!!

Tuesday, January 12, 2010

5 Card Nancy and Panel Transitions

One of Scott McCloud's more wacky inventions is the game Five Card Nancy which is based on the old comic strip Nancy. The basic premise of the game is that you can create lots of different (and fun) novel strips by combining random panels together. Scott recently posted an old collage he did that led to the game.

Of immediate note in his collage is that the sequence doesn't exactly make much sense, despite some cohesion between the panels. I'd say that it may have a narrative structure (i.e. visual grammar), but no meaning (semantics).

In some cases though, the juxtaposed panels do make sense, but the global meaning does not. In linguistics (borrowed from math), we'd call this a "first-order Markov chain", since only the units right next to each other have a connection. If a panel had a connection to two panels next to it, it'd become a "second-order chain", etc...

Markov chains were the primary way that people thought about language's grammar up until the 1950s, when Noam Chomsky then showed that grammar needed to account for connections farther than just countable individual word relationships (an approach I then applied to comics' sequences).

Essentially, McCloud's theory of panel relationships is a first-order Markov chain theory. It only looks at juxtaposed relationships. Interestingly, his Five Card Nancy game follows the same characteristic. Since players put down one panel at a time, it appears as though they are just making choices linearly. However, I'm guessing that the higher scoring combos are all ones that gel on a global scale, not just a local connection.

Also, the limitation of the panel transition viewpoint is really highlighted by McCloud's Nancy collage. How can panel transitions be correct if only local connections make sense but ones further down the sequence do not? Though we may draw and read comics one panel at time, it doesn't mean we don't build or project a bigger structure in our minds beyond the linear relations.

Monday, January 04, 2010

Storycards and visual grammar

My friend Alex sends along this link to a gift pack of "storycards". Basically, you can use these cards in sequences to create lots of different novel stories. The idea is similar to McCloud's Five Card Nancy game.

I'm interested in it for a few theoretical reasons. For example, having a stock set of units that can be combined in different ways is similar to language, where you have a set of words (vocabulary) that are combined in various ways (grammar). As the main thrust of linguistics in the last 50 years has told us, infinite possible sentences can be made with just a small set of vocabulary items, and that's basically the fun of such a card set!

However, even more interesting, its very similar to a study I finished running a few years ago and am still working on getting written up. In it, my participants were given four panels from a Peanuts strip and asked to arrange them in an order that makes sense.

People were very good at getting the original order of the strip (around 90% if I remember correctly), though that's not what I was interested in. I was more interested in the errors that people made, and whether there were patterns to them. And, indeed, there were. Prior to testing, we had coded the panels for numerous narrative properties, and found that certain narrative categories got moved around in particular patterned ways.

What this showed was that people don't just make up sequences one panel at a time as this game suggests, but that elements of that order are conditioned by roles of panels. These roles are determined both by properties of individual panels, and the relations between images.

So, one-by-one reading/drawing, but guided by underlying complexity (grammar) beyond just linear relationships.