Sunday, March 31, 2013

Help Nalini Now

This is not going to be your normal post on comics and visual language. This post is a plea for help from anyone who might read this blog and anyone compelled to help save someone's life. I summarize here much of what's said in a blog post from my colleague Sam Sommers.

From 2006 to 2012 I was a graduate student at Tufts University. One of the professors there was Nalini Ambady, who happened to be the advisor of many of my closest friends. You probably know Nalini's research in social psychology, though you may not have heard of her. Have you heard of Malcom Gladwell's book Blink, which talks about the "ability to extract an enormous amount of meaningful information from the very thinnest slice of experience"? The idea of "thin slices" comes from Nalini's work.

I am sad to say now that Nalini is battling with leukemia and is dire need of a bone marrow transplant. Because she is Indian, she has a very specific genetic marker that likely needs a match with another South Asian. Time is of the essence—she likely needs a match within 8 weeks. Her story is detailed much more at this website:

Helping save this woman's life is easy:

1) Register today as a potential bone marrow donor in the national registry. It’s easy: if you’re between the ages of 18-44 you can simply go here: 
Make sure to enter the promo code “nalini” and your request for a cheek swab will be rushed to you and its processing expedited.
2) Everyone has the potential to save a life by registering on the site. In Nalini’s case, though, it’s particularly Indian donors who are likely to be a match. Accordingly, please forward this blog post or the website to any websites, email lists, or organizations with large South Asian memberships.

Please help. It takes just a few clicks of a mouse and time is running out. Your effort may help save a woman's life. Thank you.

Sunday, March 24, 2013

New article: Visual narrative structure

I am happy to announce I have a new paper published in the journal Cognitive Science that outlines my theory of sequential image understanding. I argue that approaches like "panel transitions" cannot account for the creation of meaning in sequential images, and I offer a new theory of "Visual Narrative Grammar" that better accounts for how we comprehend visual narratives.

You can download this study along with my other papers, or directly to the pdf here.

This new theory uses narrative categories similar to traditional notions of narrative (though operationalized) that are organized into hierarchic constituents. The basics of this theory are outlined, along with diagnostic methods for testing these categories and constituents. Finally, I outline how this theory can apply beyond the sequential images found in comics to the understanding of film and verbal discourse.

This paper presents the basics of this theory for the first time in a cohesive paper, though this paper actually only consists of part of the broader theory of Visual Narrative Grammar. I hope to discuss the theory in full in subsequent papers and books. I should note also that my experiments on the cognition of understanding comics (also available for download) use this theory as their basis.

Here's the full abstract:
Narratives are an integral part of human expression. In the graphic form, they range from cave paintings to Egyptian hieroglyphics, from the Bayeux Tapestry to modern day comic books (Kunzle, 1973; McCloud, 1993). Yet not much research has addressed the structure and comprehension of narrative images, for example, how do people create meaning out of sequential images? This piece helps fill the gap by presenting a theory of Narrative Grammar. We describe the basic narrative categories and their relationship to a canonical narrative arc, followed by a discussion of complex structures that extend beyond the canonical schema. This demands that the canonical arc be reconsidered as a generative schema whereby any narrative category can be expanded into a node in a tree structure. Narrative “pacing” is interpreted as a reflection of various patterns of this embedding: conjunction, left-branching trees, center-embedded constituencies, and others. Following this, diagnostic methods are proposed for testing narrative categories and constituency. Finally, we outline the applicability of this theory beyond sequential images, such as to film and verbal discourse, and compare this theory with previous approaches to narrative and discourse.

ResearchBlogging.orgCohn, Neil. (2013). Visual Narrative Structure. Cognitive Science, 37 (3), 413-452 DOI: 10.1111/cogs.12016

Thursday, March 21, 2013

A Caveat: misunderstanding comics and the brain

Via this article I stumbled onto this dissertation which promotes using comics in educational contexts (a topic I am very interested in).

In one of the chapters of the thesis, it looks at the understanding of comics and includes a "neuroscience" section. Now, even when I disagree with them, I am always one to encourage research and writing on the structure and comprehension of comics (and I am wholly in support of the effort that this thesis is trying to make with regard to graphic textbooks). However, I have also railed on the invocation of neuroscience when used inappropriately.

Given my commitment to educating about comics and the brain, it behooves me to explain exactly what is wrong with a discussion like this, and especially what is misleading/wrong with an image like this:

This graphic—and the accompanying description—reflect the worst of misunderstandings and invocations about the brain. Here are among the problems:

1) It suffers from the "localization" fallacy. Contrary to what the image implies, there is not a "speech" location in the brain. Nor is there a "writing" or "vision" location in the brain. The brain does not localize full behaviors in the brain in modular locations. Rather, the brain is highly interconnected, with portions of the brain devoted to particular functions that interface to produce complex behaviors like speech, language, or vision.

For example, while the occipital cortex (back of the head) is the "primary" visual area, the visual system alone goes up through the top of the cortex (the "dorsal stream"), down into the bottom, temporal lobe (the "ventral stream"), and connections from the eyes go through the middle of the brain on its way to the occipital lobe in the first place. In other words: the visual system engages lots of different parts of the brain, not just back of the head. (...and where is the rest of the brain in this image???)

Reducing the brain into localized areas for each of these complex behaviors displays a lack of understanding about how the brain works (at least, to the extent we understand it so far) and for how these complex behaviors work ("language" alone can be subdivided into at least four different major substructures, which each have more substructures). Beyond this...

2) Discussing the brain is entirely unnecessary in a thesis like this. The overall point that this thesis is trying to make is that comics are very complex and involve numerous interacting parts that we understand almost effortlessly. This point does not rely on discussion of the brain to come across. In fact, I myself have done quite a lot of work describing the complexity of structure found in comics, some of which discusses the brain, and some does not.

This thesis does not need it, and including a poorly-understood discussion of the brain only hurts its overall point. Now, studies have shown that just including an image of the brain somehow convinces people that its point is more believable. However, unless the thesis is actually about the brain, such discussion is unnecessary and borderline dishonest (albeit unintentionally). Which leads us to...

3) If you're going to talk about comics and the brain, at least do the basic research to discuss what work has actually been done on this topic. Granted, there are a limited number of studies that have directly examined this issue. However, they are out there, and reviews of some of these papers can be found on this very blog: Here, here, here, and here.

I myself have now done three studies looking at comics and the brain. I'm currently writing up my latest two studies (which were my dissertation), but my first experiment is online here: Full pdf article, Short "comic" version.

I'll close by reiterating what I said in my previous post on this topic:

My point overall is this: as cool and interesting as it is, not all arguments need to be tied to the brain and cognition. And, in fact, some arguments are made weaker by doing so, since appealing to neuroscience is unnecessary at best and hand-waving at worst.

Figure out what your point is and talk about it. I'm guessing it actually has little to do with neuroscience directly.

Thursday, March 14, 2013

Hearts and bulbs

Here's another great recent Savage Chickens comic by Doug Savage. I've commented on this great strip before, and here's another good "meta"-comic:

As we all know, hearts are symbolic of love (especially over the head), while lightbulbs over the head represent inspiration. I call "over the head" symbols like these "upfixes" since they are like affixes that are "up" (a term coined by my mentor, Ray Jackendoff).

Of course, as in the last panel, a heart and a lightbulb together show a love of lightbulbs, not a love of inspiration or ideas. This combination also does not give you inspiration about love either. What's interesting about Savage's observation here is that it nicely shows that you cannot combine the upfixes together. This is a first interesting trait: these upfixes have meaning on their own, but not in combination.

Related to this, the nature of the lightbulb upfix is to give you "value added" for its meaning. It no longer is just a lightbulb, but out of the relation above the head generates a new conventionalized meaning of inspiration. By adding the heart, it effectively removes this additive meaning, making the lightbulb simply a lightbulb once again.

The heart also changes meaning a bit as well. When the heart is an upfix, it describes the mental state of the person: The chicken is in love—with what, it doesn't matter. However, in the third panel, their combination makes the heart modify the lightbulb now—it's a love of lightbulbs—not merely reflecting a general mental state of the lover (the chicken).

Finally, this combination also changes the thought bubble. In the first two upfixes, the thought bubble mostly gives a depicted link between head and upfix. It doesn't mean thinking about love or about inspiration, but just reinforces these signs as being mental states. However, it is mostly unnecessary. The heart or the lightbulb would retain their meaning without the thought bubble. In the combination though, the bubble now returns to it's usual meaning as encapsulating thoughts. Having a heart and a lightbulb floating above the head wouldn't work as an upfix, nor would it work to convey thoughts.

Altogether, this simple, quirky comic tells us a lot about the structure of these types of signs!