Tuesday, December 31, 2013

Updates and such

With the year closing, now's as good a time as any to survey some of the things going on around here…

Now that The Visual Language of Comics is out in the UK/Europe, I am eagerly awaiting its release in the US at the end of January. For those of you who are reading it, I hope you're enjoying it, and I encourage you to post a review of it on its Amazon page. I am of course always welcoming of feedback directly or posted here in the blog comments as well.

For those of you who speak German, the article about my work in the magazine Der Spiegel has now appeared online in full.

For those of you who speak Spanish, my articles have continued to be translated each month in the online magazine Revista Exégisis. The latest issue translated my article on "natural" visual language poetry (English article), and also includes example strips from my friends Alexander Danner and Tym Godek's Two for No.

This coming year looks to be an exciting one. Beyond the full book release, I continue to work on a few exciting studies here at UC San Diego that are currently underway. I should have several papers appearing in journals within the coming months, and have plans to submit several more soon. Among these projects is actually my next book, which will be an edited volume combining the work of several researchers who have contributed to this broader field that I'm working to establish (most all of whom are cited in The Visual Language of Comics). More on this as it develops...

Finally, I am very much looking forward to this coming school year here at UC San Diego. I'll be teaching a "Language of Comics" class for the linguistics department in Winter Quarter, which will focus on the structural aspects of visual language. This class builds off of the one I taught several years ago at Tufts, but now with the added bonus of having a coherent textbook in my book! Then, in Spring Quarter we'll be moving on to a class on the "Cognition of Comics" for the cognitive science department. There the emphasis will be on the experimental and corpus research analyzing visual languages, and students will be guided through doing their own research projects on visual language. I'm really excited about this class!

Here's looking forward to 2014!

Monday, December 23, 2013

Der Spiegel article

For all you German speakers, the latest issue of Der Spiegel has an article about my research. Unfortunately, the full content is not available unless you're a subscriber, but a friend of mine has translated the preview content. Here's what it reads in English…

"An American psychologist claims that pictures speak their own language, with the pictures following a fixed grammar. Does the brain perceive 'Peanuts' in the same manner as Goethe's 'Faust'?
-----------------
On the day that the pictures learned to speak, Neil Cohn didn't understand a word. In front of him lay a pile of manga comics that a friend had loaned him. He flipped through the stacks, and was bowled over by what he saw. Neil didn't speak Japanese, but he read. Picture by picture, he deciphered the story. Hours later his head hurt.

Years later Cohn worked as a comics illustrator, and now as a psychologist, and he sometimes wonders whether that day and the Mangas had something to do with what would become of him. That day, he now believes, he learned a new language; a language that has nothing to do with words."

The article is actually much longer (2 magazine pages), so if you read Der Spiegel, go check it out!

Thursday, December 19, 2013

Syntax, narrative, grammar… what!?

One of the aspects of my theories that people often find confusing is my claim that sequential images use a "narrative grammar" that is analogous to the grammar in sentences. This is confusing to many, because clearly individual images hold much more information than individual words.

This is true, but the analogy I make cuts beyond merely how much information is being conveyed, and involves deeper aspects of cognition related to their function, structure, and processing.

At the level of information, yes, panels contain much more information than words: they contain information more analogous to whole sentences. In fact, I would argue that the "narrative structure" that guides sequential images is the same structure that guides discourse and narrative of verbal stories. My theory works equally well for both of them.

So, why is there an analogy between narrative and syntax then?

First, though sequential images convey more information than individual words, narrative structure serves the same function as syntax to the communicative system. 

Both syntax and narrative function to package meaning in ways that are linear and coherent. Syntax itself is not meaning. The sentence Paul kissed Samantha is the same in meaning as Samantha was kissed by Paul, though they have different syntactic structures. Similarly, the same meaning can be conveyed in multiple ways in a sequence of images. These are the "storytelling" choices that authors make. "Narrative grammar" is the system that describes these rules.

Second, these rules governing narrative are structured in similar ways to syntax. 

There are two main aspects of this analogy. Both syntax and narrative use "grammatical categories" that provide functional roles to their units. In syntax, these categories are nouns and verbs. However, the categories for sequential images are not nouns and verbs, but rather things that I call Establishers, Initials, Peaks, Releases, etc. These categories are determined both through aspects of their meaning, but also through the ways in which they appear throughout a sequence—their "distributional trends."

Also similar to syntax, narrative is organize into hierarchic groupings. Just as syntax cannot simply move from "transitions" between one word and the next, sequential images are not understood by linear relationships between juxtaposed images. You need to be able to create groupings of images, and connect panels together across non-juxtaposed distances (such as in the above example excerpted from Tym Godek's One Night). In both syntax and narrative, these groupings then create several other constraints related to resolving ambiguity, making distance connections, coordinating the order of segments, etc.

Thus, the architecture of the "grammars" of both syntax and narrative are built in similar ways.

Third, the processing of narrative structure appears to be similar to the processing of syntax in the brain.

While this research is new and ongoing, the results so far from experiments on the comprehension of sequential images suggest that the same brain responses occur for the processing of narrative grammar as syntax (see here and here). In my experiments, I have tried to replicate the methods of classic experiments on sentence structure, and I have found similar results as these studies (short versions: here and here).

Now, does the fact that the same brainwave effects occur to syntax and narrative mean that the "same grammar" is being used in both? No. This does not mean that sequential images use the syntax of nouns and verbs. Rather, it means that the brain is treating both of these systems—these grammars—in similar ways.


For more information about my theories of Visual Narrative Grammar, check out my new book, The Visual Language of Comics, and in downloadable papers on my website.

Wednesday, December 18, 2013

Fast Co.Design article

Fast Company magazine's Co.Design site has a write up of my research over at their site.

The article focuses mostly on my "(Pea)nuts and bolts of visual narrative" article from last year (downloadable here). That was the first of my studies looking at the brain to understand the narrative grammar of sequential images, so it was fairly significant for me. Hopefully some of my more recent brain studies should be coming out over the next year!

Thursday, December 05, 2013

VLOC released in Europe!

The day has finally come…My new book The Visual Language of Comics is now available for sale in Europe! My publisher (Bloomsbury) has an interview with me over at their blog to mark the occasion.

The US release is still a little ways to go (January 30), so across the intervening time I plan to do several posts discussing some of the book. So, let's start with a little overview…

The book is divided into two main sections. Part 1 discusses "Structure and cognition" while Part 2 discusses "Visual language and the world."

The first part of the book lays out the basic structures involved in visual language:

What are the structures of drawings? Is there a lexicon of images? How do sequential images form coherent narrative sequences? What are the rules for navigating page layouts?

This first half also has a chapter summarizing the various research that has been done on cognition and comprehension. I believe that any theory about the understanding of comics should be able to be backed up by psychology experiments, both on behavior and the brain. So, this chapter summarizes this work done by me and others.

The second half of the book spends three chapters exploring how visual languages are different across the world. Visual language is not universal, and just like spoken and signed languages are diverse across the world, so are visual languages.

Why do American and Japanese comics differ in how they look? It's because they are written in two different visual languages: American VL and Japanese VL. So, there is a chapter on each of these systems, using them as platforms to discuss a variety of other issues involved in the learning and diversity of visual languages.

The third chapter in this section then provides an overview of Australian Aboriginal sand drawings—a visual language far removed from the context of comics, yet still a visual language in the sense of my theory. This chapter is especially important for clarifying how visual language is not just about comics, and it also provides a nice contrast to the systems found in comics. If we are to get at what might be "universal" about visual languages and drawing systems, then its important to look at these types of comparisons in detail.

That's the overall layout of the book… in future posts I'll try to discuss other "behind the scenes" aspects of the book's content, intentions, and preparation.

Tuesday, November 26, 2013

Violating comics... for science!

Every now and then, I see or receive commentary from people about my studies where they object to some of the stimuli in my experiments. They exclaim things like, "But, actual comics don't have sequences/pages/images like those!"

For example, in my study of page layouts (pdf), they might complain that the strange arrangements of panels (right) don't typically appear in comics. The implication of course is that doing a study that includes examples like this would not be informative because of the "weirdness" of these examples.

What these critics might not realize is the motivation for doing such "weird" manipulations...

I've had the pleasure of teaching Introduction to Linguistics at UC San Diego this quarter, and on the first day of class I showed the students how there are lots of interesting aspects of language structure that they weren't aware of, yet their brains "know" these phenomena because they are speakers of language. This creates a weird paradox, because you "know" the rules of language, but you don't have any conscious access to them. If you did, linguistics as a field wouldn't exist!

"So," asked one of my particularly astute students, "how is it that we can study this stuff if we don't have conscious access to the rules?"

The answer, I said, is by violating that structure. If we create bad examples of language, then it can tell us about the constraints involved on language that make such productions ungrammatical. For example, I can say He likes her or She likes him, but not *Him likes she or *Her likes he (*=ungrammatical). These latter sentences should sound like garbage! Yet, these violations provides us with evidence that different pronouns are used for the subject and object positions of sentences (nominative vs. accusative case), even though they essentially contain the same meanings (he/him = masculine noun, she/her = feminine noun). By violating the structure, we can figure out the rules.

The same principle applies to studying the visual language used in comics. By manipulating and violating the structure, we can see people's reactions and thereby deduce the rules and constraints that might be operating on those structures.

So, it is true that most of my manipulations to stimuli wouldn't be found in "actual comics," but that's exactly the point. People don't generally produce things that truly violate the constraints of the structure.  However, by doing such violations we can learn about how that structure works and is instantiated in people's minds and brains.

Sunday, November 24, 2013

Guardian Article

David Robson discusses my research and my upcoming book, The Visual Language of Comics (out in less than 2 weeks in the UK!) in a new article for the UK's Guardian newspaper.

He interestingly ties my work to aspects of cave paintings, which I've discussed on this blog just a little. However, the article does a very nice job of summarizing a lot of the aspects of my research program. Go check it out!

Monday, November 11, 2013

Review: Indonesian manga analysis

I recently found this interesting paper, "Impacts of manga on Indonesian readers' self-efficacy and behavior intentions to imitate its visuals," that attempts to dissect the factors influencing why Indonesian comic fans imitate the drawings in manga (pdf at academia.edu...account may be necessary to access).

They gave readers a copy of a manga (Naruto) and a non-manga comic (Tintin) and then had them rate the attractiveness of their visuals, the intentions behind imitation, and other factors. They found that people rated the manga as more attractive, more engaging, and eliciting more "psychological" responses. They conclude that "emotional attractiveness" is the primary factor for why people prefer manga to non-manga comics.

First, let me say that I really like that people are doing these types of sociological studies looking at these issues. I like the overall aims of this study, and especially the use of a data-driven methodology, and would like to see more approaches like this.

However, I think that the actual results are pretty confounded. The authors admit, for example, that the participants in the study already read comics, particularly manga (and some even had already read this issue of Naruto, but not Tintin!). This means that the results aren't actually finding information about "blank slate" preferences for some inherent quality of manga vs. non-manga (here, a European comic). Thus, the results are a bit confounded for the intent of the study. So, this study doesn't necessarily tease apart the influences on why they like and imitate manga.

Rather, these opinions reflect participants' tastes having already selected manga as having their visual language of choice. Given this, I think the results more provide evidence that people who are already have preferences for comics using a particular visual language will therefore deem it more positively than comics using a different visual language. This might seem fairly trivial (of course people like the things they already like!), but providing in-group vs. out-group effects for a type of visual language would be consistent with the same effects that occur in spoken languages, where people have more positive views of their own dialects to others.

This of course leaves open the question of "why?" people are so keen on imitating manga across the world (Indonesia included). I personally think there are many factors, including sociocultural factors (the "coolness" of Japan and/or the types of people in the new country that read them), economic factors (price of books, etc.), story factors (subject matter, differing genres etc.), and others. However, as I have argued in several papers and my upcoming book, I think there's also a cognitive factor based on the consistency of the visual vocabulary. Essentially, since the same visual language (i.e., "style") is used across most all the books, it creates a consistent template for people to imitate. Compare that with the relative diversity in American and European comics—it's much harder to identify a "group style" to associate with (and thereby become an in-group member of that "visual linguistic community").

So... while I don't think that the results really support what they set out to look at, I think this is an interesting paper nonetheless and I'd like to see more approaches to analyzing these sorts of issues using similarly data-driven approaches.


Ahmad, Hafiz Aziz, Shinichi Koyama, and Haruo Hibino. 2012. Impacts of manga on Indonesian readers' self-efficacy and behavior intentions to imitate its visuals. Bulletin of JSSD 59 (3):75-84.

Monday, November 04, 2013

Tufts Magazine

The latest issue of Tufts Magazine has a nice write up about my talk from this year's Comic-Con. Go check it out!


Tuesday, October 29, 2013

Coming soon...

My first printed copy of my upcoming book arrived in the mail today!


The book looks beautiful inside and out, and it makes me even more excited in anticipation for its release very soon: December 5th in the UK and January 30th in the US, though preorders are already available.

As the release date(s) get closer I'll start posting more about the contents and what it covers. Stay tuned!

Sunday, October 27, 2013

New Paper: Beyond Speech Balloons and Thought Bubbles

I'm pleased to announce that my paper, "Beyond Speech Balloons and Thought Bubbles," (PDF) has now been published in the journal Semiotica. The paper dissects the ways in which text and image connect to each other. I argue that speech balloons and thought bubbles are actually two "surface" representations of a deeper structure where the meaning can be varied by just a few primitive variables. The overall discussion then reveals a lot about the elements involved in text/image relations, their connection to panels and other aspects of the visual language of comics.

This paper has actually been through a long journey. The first draft of this paper was actually posted on my website way back in 2003—ten years ago! I then expanded and refined it into its current form in 2009, and after pulling it from the journal where it was originally accepted (and languished for some time without being published), it finally found a published home. So, I'm glad it's now finally out!

The full paper can be accessed directly here (pdf), and is of course available with all my other papers.

Full abstract:
Speech balloons and thought bubbles are among the most recognizable visual signs of the visual language used in comics. These enclosed graphic containers provide a way in which text and image can interface with each other. However, their stereotypical meanings as representing speech or thought betray much deeper semantic richness. This paper uses these graphic signs as a platform for examining the multimodal interfaces between text and image, and details four types of interfaces that characterize the connections between modalities: Inherent, Emergent, Adjoined, and Independent relationships. Each interface facilitates different levels of multimodal integration, tempered by principles of Gestalt grouping and underlying semantic features. This process allows the possibility of creating singular cohesive units of text and image that is on par with other multimodal interfaces, such as between speech and gesture.

Cohn, Neil. 2013. Beyond speech balloons and thought bubbles: The integration of text and image. Semiotica. 2013(197): 35-63.

Tuesday, October 15, 2013

Review: Comics and Language by Hannah Miodrag

Among the many books on "comics theory" published this year, Comics and Language by Hannah Miodrag may be the most relevant for my own interests. It directly analyzes the relationship between comics and language, especially proposals that "comics are a language." There are many complex aspects of this work, and I could easily spend great lengths discussing the minutia in every chapter. I'll instead try to hit upon the biggest points in this review as I see them.

The book is divided into three sections. The first tackles the actual language in comics, often pointing out that it has been under-appreciated in research. The second section looks at claims (like my own) that sequential images constitute linguistic-like structures. The final section analyzes how the component parts of images may or may not be like languages.

Overall, I found it a curious read, being one of the more well-reasoned and better written works on comics (a rare treat), yet at the same time being deeply flawed. On the one hand, she often makes insightful and important clarifications showing a clear understanding of many of the underlying concepts she discusses. For example, she expertly applies Groensteen's theories and my own (older) theories to the analysis of several works—and does so with aptitude. Rarely does it seem that she misunderstands any of the theories of comics she discusses, and several of her critiques could have a profound impact on the field of "comics studies" at large. On the other hand, the work also contains glaringly huge problems that undercut the foundations of many of her broader arguments.

Altogether, the book is potentially an important read for a burgeoning "comics studies," but must be read with the understanding of some caveats.

Disentangling comics and language

The surface statement of the book argues against the idea that "comics are a language"—be it the graphic form itself (what I would call "visual language") or the combination of images and text. She also argues that the use of verbal language in comics is significantly under analyzed, and that analysis of the images could benefit from art historical treatments discussing style.

Like many works arguing against comics being like language (and even some arguing for this comparison), Miodrag does not entirely tease out the issues involved, even if she understands them. For example, she acknowledges and understands my idea of a "visual language" that is used in comics (and, with rare praise on her part, likes the iteration of my grammar she discusses), but does not describe how it changes the argument with regards to the comparisons of comics and language broadly. My argument is not that "comics are a language" or "comics are like a language." Rather, it's that comics are written using two processes: writing and drawing, and that the structures of both of these are similar (that is, the visual language created by drawing is similar in structure to the verbal language created by writing/speaking).

Miodrag persists instead to treat "comics" as a unified object—not a symptom of its parts (writing and drawing) embedded within a certain sociocultural context—and therefore maintains comparing "comics" and "language" throughout, as if the issue were not more complicated than that. I can see two possible reasons for why she maintains this stance. One is that the more nuanced argument is one solely made by myself, and Miodrag would, understandably, rather argue against the shortcomings of the broader field as a whole, thereby sidestepping my claims rather than seeing them as a triangulation of these issues. A second reason though could be that this orientation lies in the desire to discuss comics both as art objects (i.e., works of literature/aesthetics) as well as to discuss how they work (i.e., how the structure works). This relates to a broader point underlying the book, which I'll return to later.

Analyses and insights

The best parts of the book are when Miodrag analyzes actual comics, whether applying particular theoretical concepts to these discussions or not. In particular, she excels in her descriptions of the creative use of verbal language in a variety of comics. For example, her demonstration of Herriman's playing with the sounds and meanings of words and phrases in Krazy Kat is both insightful and compelling.

To this broader issue, Miodrag importantly points out that literary analysis of comics may not emphasize enough the skills of writing in the text of comics. She argues that the notion of "good writing" in comics often overlooks the actual artistry of wordplay in favor of lauding wider notions of storytelling and interesting plot ideas. She aims at Moore's Watchmen in particular, which is lauded as a masterpiece, yet Miodrag convincingly finds it a bit lacking in the contribution of its prose. This broader critique is certainly not made very often (though, apparently was recently... were they listening to her??), yet it is hard to deny the necessity for analysis of text in an artform that uses both verbal and visual languages. This may be my favorite insight and advocacy of scholarship she raises. A parallel argument is also raised about integrating art historical viewpoints into "comics studies," a stance with equal logic.

She also nicely points out that not all comics should be considered as "literary" or "artwork," just as not all novels are considered to reach that level. This is another rare statement of honesty in comic theory, since the defensive "comics are just as good as all that other stuff" viewpoint often casts the net too wide in claiming that "comics are art/literature," without acknowledging the wide variance of quality.

Misunderstanding language

In contrast, Miodrag is most out of water when discussing the structure and properties of language itself. Unfortunately, this orientation permeates the entire book. Here, the problem is one of paradigms: Miodrag's conception of language uses "semiotic" notions of language (structuralist, post-structuralist). Within this paradigm, Miodrag's critiques make sense and are well-reasoned. However, this view has long been rejected from the dominant thinking of how language works since the "cognitive revolution" in the 1960s (and the subsequent advances made since then), making most of her assumptions about linguistic mechanisms outdated and/or wrong.

For example, Miodrag bases her criteria for what constitutes "language" on things like the use of arbitrary and discrete signs which use minimal units. She apparently gets these criteria from a 1986 book by J.T. Mitchell (not a linguist), though these criteria actually come from a well-known paper by the linguist Charles Hockett in 1960 (uncited by Miodrag) where he proposes "design features" of language to distinguish it from the systems of communication used by animals. However, this list of features predated the cognitive revolution by many years, and failed to be updated by the insights that this paradigm shift allowed. Furthermore, many of these criteria are thrown into doubt from the research on sign language, which also appeared long after Hockett's list (and which sign language research often had to fight against).**

Thus, for a book that is centered around disentangling the claims about the structure used in comics being likened to language, it is woefully under-informed about contemporary notions about what language is and how it works. In fact, the only recent works of linguistics research that are cited (such as Pinker's Language Instinct) are surprisingly mentioned in contexts outside framing how language works. Rather, citations pointing to expertise on language rely largely on the semiotics literature (often fairly old) and other sources outside the authoritative fields that study language.

This is a large oversight, especially given that the book otherwise appears to be well-researched, logically organized, and contain genuine insights. If claims are to be made about language with any authority, then it is not unreasonable to expect that contemporary notions of language be consulted and used to address the issue. Without this, many of Miodrag's doubts of the linguistic status of elements in comics fall flat because they are grounded in a view of language that is widely accepted as outdated.

Although, to be fair, most of the works claiming "comics are language" that Miodrag argues against are also outdated and/or under-informed, coming from the same semiotic perspective. Indeed, Miodrag's larger point is to argue against the notion of "comics as language" embedded within this semiotic framework specifically. In this criticism she is correct, though not entirely for the reasons she thinks. Lacking in this though is an actual critique of comparing comics to what we now know about language. So, her argument against "comics as a structuralist type of language" rings true, but the broader question of "comics relationship to language" is ultimately not addressed. I see recognizing this distinction as crucial for getting the most benefit out of Miodrag's work, even if the book does not intend it.

Structure vs. Literature

The orientation towards semiotics as a paradigm in general is what complicates Miodrag's work the most. Pervading the whole work is tension in whether Miodrag is trying to discuss how comics work in a structural, cognitive sense—i.e. how is the medium structured and how do minds comprehend it—or how comics can create "meaning" as works of art/literature (where appropriate)—i.e. what are the techniques used to create artistry.

Miodrag appears to conflate these questions, as does much of "comics theory" with a semiotic orientation (including Groensteen, Postema, and others). This is quite apparent in her advocacy for the idea that comics be understood as a "network" rather than a sequence. The "network" idea that each panel connects with every other in a comic is most often attributed to Groensteen, and is one that I summarily rejected on the account of it being entirely cognitively unfeasible. Miodrag in fact admits the logic in my counter-argument against this position, yet defends it as a worthy idea to hang onto nonetheless because it allows a discussion of the "aesthetic" features of a comic.

Now, I do not think that Miodrag holds onto this idea irrationally. The whole of the book shows just what a quality thinker she is. Rather, I think she does it because she ultimately struggles with separating the study of comics as aesthetic art/literary objects from the basic understanding of the mechanisms that underly their comprehension, here couched in the "comics as language" comparison (or as "formal theory"). Yes, the network idea is unrealistic when viewed from a cognitive perspective concerned with comprehension. But, if the desire is to describe how comics work as "literature" or "artwork," then such basic cognitive understandings may not be necessary to discuss what is important at the level of aesthetics and "artistic message" (though I might argue that those cognitive distinctions can better inform aesthetic analysis, but that's beside the point in this case).

Miodrag does not outright advocate for this separation between cognitive understanding and artistic interpretation, though she does at least acknowledge that such ideas exist on separate planes. After reading the book though, it is clear that such a separation should be acknowledged more forcefully throughout scholarship on comics.

"Comics theory" as literary analysis should not attempt to explain how they work in any cognitive or structural sense any more than literary treatments of novels should try to explain how language works. Research on how language works has distinct fields already: linguistics, psychology, and cognitive science—and it is those fields that should be the ones to study how the visual language used in comics operates as well. "Semiotic" theories like Groensteen's (and Miodrag's and Postema's) often only confuse and muddle the waters of this distinction.

Thus, if the take home message of Comics and Language is that "comics studies" should focus on the capacity of some works to be artwork/literature and described in terms of their own characteristics—leaving behind describing their formal structure (or at least leaving it to the cognitive sciences)—then I fully agree with these conclusions, even if they take a circuitous and bumpy route to reach.

-----------------------------

Notes:

** In my own upcoming book, I directly address the "design features" of arbitrariness, minimal units, and a visual vocabulary from Hockett that Miodrag finds troubling. This review is already long enough as it is, so I felt directly addressing here how the cognitive viewpoint differs from the structuralist viewpoint would be too much. If people want these arguments though to be made on the blog, let me know and I'll cover them in subsequent posts. As always, my recommendation for learning about actual contemporary linguistic theory is Foundations of Language, by my mentor, Ray Jackendoff.

Thursday, October 10, 2013

Programming with Grawlixes

Marc van Elburg has sent along a paper he's written about "grawlixes," which are the random symbols used often in comics to omit profanity, like $#!+! You can read his essay on it here.

He follows many of the categories proposed by Mort Walker's Lexicon of Comicana, though he departs a bit from that as well. His discussion also expands from grawlixes specifically to discuss aspects of the "visual language lexicon" such as motion lines and others. He then expands the discussion beyond what is found in comics, to exploring how these signs could undergo additional rules to make them more systematic.

It's definitely an interesting read. I don't completely agree with his categorization and conclusions in their entirety, but I like the approach of systematically cataloging various graphic signs and testing how they work in various contexts. As I argued before, I think that these are the only ways you can really get at understanding how the system of the visual language used in comics works.

Edit: And curiously timed, here's an article by the linguist Ben Zimmer on the origins of grawlixes!

Monday, September 30, 2013

Review: Narrative Structure in Comics by Barbara Postema

Barbara Postema's new book, Narrative Structure in Comics, joins a nice list of books on comic theory to emerge this year, and it provides some interesting analyses and insights. The book is clearly written and should be readable for most audiences.

Upfront, I found it unclear who the book is aimed at. If this is an introductory book or for a popular audience (perhaps suggested by the appendix of basic terms at the end?), some of the terminology and ideas should be more carefully laid out and should reflect or acknowledge the consensus of the field. If this is a scholarly tome, some of the background literature should be more carefully acknowledged and debates discussed in detail.

For example, quite pertinent to my own interests, she quickly dismisses analogies between the structure of comics and the structure of language, stating that “…images communicate largely without rules… the smallest elements of images have no set meanings, and the way these elements are combined or even repeated are not governed by rules like grammar” (p. xvi). Viewpoints like this are repeated throughout the text, stated as accepted knowledge or as obvious, despite the absence of a scholarly consensus on the issue or any evidence to support it. (Compare this to Miodrag's recent Comics and Language—review coming soon—which spends 200+ pages discussing this argument!)

In fact, comparisons have been made along the lines she disparages since the 1970s within "semiotic" approaches to comics—the viewpoint she takes throughout (see my review "Comics, Linguistics, and Visual Language")—which go unreferenced. I assume her argument is directed against these researchers' theories, but it's hard to know without references or augmentation. My own work highlighting parallels between language and comics with a contemporary linguistics/cognitive viewpoint also go undiscussed and unreferenced throughout. It's hard to know whether such omissions are by choice (then: why?) or out of ignorance (which I doubt).

This again makes it unclear who the audience is: are readers supposed to take such flippant claims as reflecting the consensus of a field? Are they supposed to know that they reflect an opinion of a particular group of people (or at least, this author), but that there is an ongoing debate about this issue?

Clarity in this regard would aid the book on the whole, even if made as a note in the introduction, and would certainly aid me in reviewing it. I'd be more harsh on the book if it's meant as a scholarly treatise than if it's meant as an introductory book. I'll strike a middle ground here.

Theoretical contribution

This issue of audience permeates my main problems with the book, which is that I did not see much novelty in the theoretical contribution. Her broadest argument is that every level of structure in comics involves "filling in gaps" of information that go unprovided in their representations. This applies to the inferences about objects that are made by viewing only the simplistic forms of cartoony images as well as to the inferences created about a sequence by viewing only what is depicted in panels.

Certainly, no one denies that inference occurs in the understanding of sequences of images (or individual images), and indeed many people have said this, including McCloud, Saraceni, myself, and a host of others writing about comics (and other types of narratives!). Just how her theory of inference goes beyond others in any substantive way remains largely undiscussed. Is this a reflection of the book being aimed to an introductory audience? It's hard to tell.

A comparison may be useful: Saraceni's dissertation was extremely detailed, technical, and precise, only to be overly-simplified in the "textbook" version that was widely published as The Language of Comics (his articles fared better, striking a nice balance in tone). I haven't read Postema's dissertation, but such a simplification may be the case here as well.

In some ways, this book feels like a meshing of McCloud's focus on inference with a desire to simplify, clarify, and in some cases go beyond Groensteen's ideas, but aimed at a wider, less academic English-speaking audience (thankfully with less opacity than the original). For example, her contribution to discussing page layouts is merely to categorize the ways in which panels might be physically juxtaposed: do they have big gutters, no gutters, inset panels, etc? This focus on the physical is comparable to Groensteen's taxonomic treatment of how word balloons relate to panel borders (do they touch, do they overlap, etc). Such analyses at least provide a way to cut up the taxonomic space of comics, but they don't tell us much about their understanding. They're just a list of things in comics—and not even an all that interesting or insightful one!

Also, I certainly don't believe that all of these variations carry the intrinsic aspects of "meaning" that Postema seems to argue that juxtapositions in layout carry with them. For example, she makes much of the difference between when a single line separates panels versus having a larger gutter. However, when people first start drawing comics (for example, as kids and often adults), they usually just use single lines to separate panels. Do the "meaningful" "significations" of a gutter versus a line maintain in these, non-published works as well?

Semiotics vs. cognitivism

This example highlights the nature of a significant portion of my critique of the theory, no matter the audience. That is, the dissatisfaction with the book's orientation within a "semiotic" paradigm—a trend I'm finding in many recent works on comics and which I do not consider to be useful for looking at the formalist elements of this medium. These ideas draw from linguistics concepts from the 1930s until 1950s, principles of which were borrowed by the social sciences and humanities for looking at many other non-language domains. However, the actual study of language (i.e., linguistics and psychology) largely abandoned this approach in the 1960s in favor of ideas that attempt to describe principles of cognition, for which those "semiotic" ideas were not useful and only described a surface structure. Since that time, cognitive science research has developed enormously in ways that are insightful for describing language and beyond.

Though Postema appears to argue against a structuralist idea of language, she still maintains the broader framework of semiotics. There are many places where notions from contemporary linguistics or cognitive science would provide more convincing frames for the analyses of comics she makes (which, admittedly, are often insightful and interesting despite their non-cognitive framework). For example, Postema's discussion of Jacks Luck Runs Out by Jason Little nicely highlights how the story draws inspiration and structure from the rules and aesthetics of playing cards. However, her discussion remains at a surface level, stated in terms of connotation and denotation as if such principles carry with them explanatory power of "understanding."

In terms of cognitive science, this comparison would be covered by the theory of "conceptual blending," which illustrates how one domain of knowledge can creatively draw from another domain of knowledge. Blending theory would actually be beneficial for many of the examples discussed throughout. Granted, Postema's insights would likely remain much the same if framed in terms of blending, but framing them as such would allow her to probe deeper issues at work in this analysis, as well as to connect such a discussion of comics to how we understand the workings of the mind and brain more generally (and, dare I say it, language!).

As it stands, the analysis remains at surface level, without casting much light on what might be going on "under the hood" in the actual understanding of this material. This is especially true of the broader claim that readers "fill in gaps" when reading comics: Just what is creating this inference if not the mind/brain? And if it is the mind/brain (which I doubt Postema would deny), then why not actually target those principles guiding such processes?

Here is where the semiotic approach is most clearly lacking from cognitive approaches. It is important to remember that the "comic medium" itself does not actually contain these structures, as purported by the semiotic approach. Rather, the human brain is what instills and interprets such structures within the graphic form, so that is where our research should be focused in such matters.

Wednesday, September 18, 2013

Corpus analyses of comics

I stumbled across this interesting project called the eBDtheque database which has started coding various comic pages to create a searchable corpus. The full project is online here, while their paper describing the project is found in this pdf.

This rather large research group looks like it's actually been doing quite a lot of computational approaches to analyzing comics, which is very cool.

Besides recording background information about the authors, publishers, etc. of the comics, this database in particular seems to take comic pages and code them across three main dimensions. 1) How many panels per page, 2) how many balloons per page, 3) how many lines of text within balloons.

While this coding scheme is fairly limited, they have said that they'll be expanding it to look at other dimensions, such as the angle-of-viewpoint and filmic shot type. What makes the project fairly impressive though, is that the data in this corpus is not just human coded, but also involves extraction from the computers themselves. All that makes this a project worth watching.

Naturally, I'd love to see this type of project be developed further. My two cross-cultural studies so far comparing panels coded from Japanese and American comics are hints of what having a corpus will allow.  I have a research library so far of roughly 4,000 comics and graphic novels that is ripe for research to be done on them. The compelling idea here though is not just doing isolated studies. Rather, the idea would be to create a massive database that could be searched in many different ways for various inquiries across many dimensions. In fact, I had talked with programming savvy friends about starting such a project years ago.

Once my own actual lab is up and running, I'd like to start on a project like this by recruiting students to help with the coding. However, another way to build a large database is to "crowdsource" the work. This would involve having people across the world complete "training" sessions to become proficient at the theories and coding schemes involved, and then they could login and code comics they have around them.

This would help with creating a large, internationally diverse corpus available to many scholars to use, and would be created by people across the world. Imagine being able to do large scale searches across millions of comic panels from across the world in order to do comparisons about the structure of visual language!

Thursday, September 12, 2013

Methodological advice for comics scholarship

In my previous post, I drew a distinction between the type of scholarship often done by people who are creators of comics (i.e., fluent in the visual language they study) and scholarship by those who lack this fluency. I argued that there are certain things that non-fluent scholars are unable to recognize, and that such impairments can significantly limit their viewpoint on studying the structure of sequential images.

However, there are ways in which these limitations can be overcome and significant contributions can still be made. Much of this comes down to methodology (important also for those who are fluent creators as well), which is a more important critique I often have about scholarship on comics:

1) Be specific — Sweeping generalizations about "how comics work"—especially those based on specific exceptional examples—are almost of little contribution in understanding the structure of the visual language. Papers in linguistics or psychology almost never try to generalize to "all of language" or "all of cognition," but rather pinpoint a specific structure and then rigorously detail it. Scholars of the structure used in comics will be far better served by finding a very specific instance of something of either 1) theoretical or 2) empirical interest and then rigorously investigating it.

1.1 - Things of theoretical interest might be a particular idea or theory about the structure of sequential images, such as "how panel transitions (don't) work" or "how panels structure space" which can then be rigorously examined across many comics—both "normal"and "artistically exceptional." It always amazes me that so many scholars subscribe to McCloud's theory of panel transitions, yet practically none of them follow his methodological lead in trying to apply those transitions in quantitatively in the way that he tabulated transitions in various comics. Every time I did this, it directly poked holes in the theory of transitions.

1.2 - Things of empirical interest would be seeing a phenomenon in a comic, and then rigorously analyzing how it operates in other books (if it does) and exploring the theory behind why it might occur. A good example of this method is Abbott and Forceville's examination of why some character's hands turn into stumps in a manga. They noticed a curious phenomenon, then detailed why they thought it—specifically—happened.

2) Manipulate and test — Following the need for specificity, you should then manipulate structures and test them on people. If you aren't fluent in the visual language and can't just manipulate it in your mind (or even if you can...), manipulate things based on assumptions of your theory, and then give those creations to people who are fluent.

For example: I've often seen people claim that "sequences in comics can't just be strictly linear" or "sequences/narratives can't be governed by a rule system like a grammar because nothing constrains the sequences in such a way." Yet, not one of the papers I've read that claims this ever provides actual evidence supporting such critiques (compared to, say, my psychology experiments which directly test this and say otherwise).

Here's what they should do with the most simple of manipulations: take a sequence of images that they claim "isn't constrained" and then scramble the images into random orders. If every possible order makes sense, then they might be right that there are no constraints. If even one order doesn't make sense, then 1) there are constraints, 2) their critique/hypothesis is wrong, and 3) it then behooves them to follow up and figure out why that one order doesn't make sense (i.e., figure out the constraints!).

Even if you don't have the intuitions to create the sequences by drawing them, you can still manipulate existing sequences based on a thesis (again, of a specific phenomenon), and should have the intuitions to notice if what you did has an effect. Similarly, you can show many manipulated sequences to other people and start to derive data that tells you which manipulations might be behaving in different ways.

...and congratulations, at this point you're doing science!

Thursday, September 05, 2013

Comic creation expertise and theory building

When I first started entering into discussions about visual language research, the fact that I actively created comics seemed like an important point that I would often stress. While I am a little more forgiving about non-creators doing theoretical work these days, I think there are some interesting trends that fall out of whether a person actively has created comics or not.

People who create comics, especially at a professional level—like Eisner, McCloud, and myself—tend to focus on the fairly normal, banal, and ubiquitous elements of the medium. This research usually emphasizes regularities, showing how things are systematic in unexpected ways, and/or providing taxonomies to better understand the pervasive and everyday elements of this visual language. 

On the other hand, people who do not actively create comics tend to theorize or discuss about the exceptional, unusual, and "rule-breaking" aspects of comic creation. They discuss the most artistic and poetic creators, and their examples usually are the most "interesting" ones out there. They are also the most likely to disparage the idea that regularities do exist in visual sequences.

This distinction makes sense to me. To people who are not actively immersed in the visual language, the things they find most interesting are the things that break the rules. Normal, ubiquitous elements of the system are just that: normal. They are less interesting because they are commonplace and less noticeable. These people also have less intuitions for the nature of the regularities, being unable to manipulate such structures in their own minds (thereby giving the illusion that no such structures are there).

For actual creators, the banal is highly interesting, because it allows us to understand and articulate the basic process and the intuitions that we have while creating. Our scholarship can be informed by how we actually think, as opposed to just what we see on a page far removed from the creative process. The medium is not a static, received thing, but rather a dynamic, created process.

I feel that this distinction is fairly important. As I've said before in my "advice to aspiring theorists," there is a danger in placing too much focus on the exceptional examples. Exceptional examples can be useful for revealing what makes them exceptional—and thereby highlighting how they contrast from the normal. However, building a theory entirely around the exceptional will leave out the normal, and it will ultimately be left unable to deal with the most basic aspects of the medium.

Tuesday, August 27, 2013

Full year of comic theory books

2013 seems to be quite the year for books about theories of the comic medium! There've been at least four volumes that have come out this year, plus my own, The Visual Language of Comics, which is due out in December.

We started earlier in the year with Thierry Groensteen's Comics and Narration, the anticipated follow up to his The System of Comics. I planned to do a full review of this new book, but after awhile, I realized that there wasn't much I had to say that wasn't already articulated in my review of System of ComicsYou may recall, I was not overly impressed. On the plus side, the writing in this new volume is much better—whether this is attributable to Groensteen or his translator, Ann Miller, is unknown, but either way it's a great improvement that should be lauded. Beyond that, my previous critiques with regards to the content all still stand.

More recently, we've seen the publication of Barbara Postema's Narrative Structure in Comics and Hannah Miodrag's Comics and Language. Both books take umbrage with the idea that the visual language in comics can be structured like language. Miodrag actually devotes the whole book to arguing against this view! I will likely post focused reviews of both these books in the coming weeks/months.

A similar viewpoint is taken by several of the chapters of the recent collection From Comic Strips to Graphic Novels, which features a wide range of articles about comic theory, all from perspectives in the humanities. I may or may not make posts discussing all or some of the chapters, since I only recently began reading it. As is often the case in big collections like these, some of the chapters seem much better than others.

Outside of Groensteen's books, it's actually surprising how nearly all of these books have some sort of disparaging viewpoint on "linguistic"-styled approaches to describing the structure of sequential images. Among these, Miodrag's Comics and Language is the only one that is actually grounded in any sort of logical and reasoned argument in this regard (though it too has some limitations). The rest seem to have no idea what language is, how it might work, or how a "grammatical" system would or would not behave (though, this goes for Groensteen too—and he actually does claim comics are like language!).

I plan on addressing many of these concerns in my focused reviews for the books, but my broader response will appear in my own book at the end of the year.

Opinions aside, it's nice to see so many books emerging that attempt to broach the topic with seriousness. Now we just have to work on the approach...

Thursday, August 22, 2013

New paper: Prediction, events, and the advantage of Agents

I'm excited to say that I have a new paper out in the journal Cognitive Psychology with my good friend Martin Paczynski! This one looks specifically at the structure of events in visual narratives, particularly how people process characters acting as agents or patients.

As with all my papers, you can find a link to it on this page, or a direct link to the pdf here.

Here's the abstract:

Agents consistently appear prior to Patients in sentences, manual signs, and drawings, and Agents are responded to faster when presented in visual depictions of events. We hypothesized that this “Agent advantage” reflects Agents’ role in event structure. We investigated this question by manipulating the depictions of Agents and Patients in preparatory actions in wordless visual narratives. We found that Agents elicited a greater degree of predictions regarding upcoming events than Patients, that Agents are viewed longer than Patients, independent of serial order, and that visual depictions of actions are processed more quickly following the presentation of an Agent vs. a Patient. Taken together these findings support the notion that Agents initiate the building of event representation. We suggest that Agent First orders facilitate the interpretation of events as they unfold and that the saliency of Agents within visual representations of events is driven by anticipation of upcoming events.

ResearchBlogging.orgCohn Neil, & Paczynski Martin (2013). Prediction, events, and the advantage of Agents: The processing of semantic roles in visual narrative. Cognitive psychology, 67 (3), 73-97 PMID: 23959023

Monday, August 19, 2013

El Lingüista Ilustrado

Hey Spanish-speakers! I'm excited to announce that some of my articles will now be translated to Spanish in an ongoing series in the digital comics magazine Revisita Exégesis starting with the current issue #24.

The magazine features several great original short comics by various authors, articles, interviews and a lot more. You can view the issues online or download them as pdf or cbz files.

My contribution—"El Lingüista Ilustrado"—reprints of several of my blog posts and articles from my website translated into Spanish for the first time by the editor Simud. Different articles will appear with each issue.

It's quite an honor to have them translate my works to Spanish in such a venue, so go check it out!

Friday, August 09, 2013

Eurotrip 2013

I've just returned from an exciting trip at the Cognitive Science Society conference in Berlin. My presentation was a brief version of my dissertation work (amusingly, roughly the same material from my talk at Comic-Con two weeks prior). My dissertation was actually awarded the Glushko Dissertation Prize this year—named (and funded by) the ever-enthusiastic Robert Glushko.

Even more exciting than that though, at the conference it was announced that my mentor, Ray Jackendoff, will be awarded next year's Rumelhart Prize! This award is essentially the "Nobel Prize" of cognitive science—the field's most prestigious and highest honor. So, I'm incredibly excited and proud that it's being awarded to Ray next year, and I can think of nobody more deserving.

After the conference in Berlin, I spent a few days over in Amsterdam. While I was there, I not only got to experience the fantastic comic book store Lambiek, but had a great time finally meeting Charles Forceville.

As I've mentioned on the blog before (here and here), Forceville is one of the few other scholars who has looked at the overlaps of contemporary theories of linguistics to the visual language of comics. Specifically, his work examines how images and multimodal interactions involve underlying conceptual metaphors—mappings of one conceptual frame onto another

If you're not familiar with his work already, I highly recommend checking it out (this article is a good place to start, as is his edited book, Multimodal Metaphor). Hopefully he and I will be working on several projects together in the future.


Friday, August 02, 2013

Scott McCloud and the scientific method

After my talk at Comic-Con I was delighted to be able to chat with Scott McCloud for awhile. He actually came to my talk, and I think it was one of the first times he's ever seen me present my research live, despite our having known each other for about 15 years now.

One of the things we talked about was the use of scientific method to study the visual language of comics. My approach of course is based on a scientific view of this research.

McCloud's work often gets flack from various people within academia for not being theoretically rigorous enough, being ignorant of "the literature" that came before it (mostly outside of English), and for essentially not being "scholarly" enough.

However, I think that McCloud's ideas—as basic as some of them are—are perhaps the most scientific approach to comics up until that time. In this way, this work distinctly separates itself from other theories, like Groensteen's System of Comics, which are distinctly un-scientific.

What I mean is that McCloud's theory is based out of a desire to understand how certain aspects of comics were different in the world. Unless I'm mistaken, McCloud's theories greatly grew out of a desire to explain how the things he was seeing in Japanese manga were different from those in American comics. This was the observation that provided a problem, the theory provided the solution.

But, here's the key thing: he then used his theory as a tool to actively quantitatively analyze the properties of actual comics. This is especially salient in his cross-cultural comparisons of panel transitions. He actively used his categories in a useful way to 1) confirm the hypotheses of his original observations, and 2) to illuminate aspects of the medium in interesting ways.

This was what was so novel and interesting about McCloud's book when it came out. Regardless of whether he got the actual theorizing right, or if his work paid lip service and connected to a scholarly tradition, the most important insight was in its methodology. Very few approaches to theories about the visual language of comics have done this prior to McCloud, and very few other than mine have done it since.

On this point, I should mention that my work has largely grown out of this same trend. Besides the empirical experiments and data collection, my theory has grown by looking at actual examples and then applying the theory—seeing where it goes wrong—and then revising the theory.

In fact, this is how I first noticed that I needed to expand the number of transitions beyond McCloud's six, and eventually how I realized that transitions did not work and that an alternative approach was necessary: I was trying to apply McCloud's transitions to actual comics using the same type of quantitative analysis that I found in Understanding Comics.

Monday, July 22, 2013

The Chinese Room

A few months back I got a request from my friend and colleague from Tufts, the philosopher Dan Dennett. Dan is the co-director of the Tufts Center for Cognitive Studies along with my mentor, Ray Jackendoff, so I got to benefit from his wisdom and humor throughout graduate school.

He asked me to recreate a comic strip he saw posted on the wall of an MIT Artificial Intelligence lab almost 30 years ago by an unknown author. The comic is based on the philosopher John Searle's famous thought experiment about the "Chinese Room."


An "official" posting is also found on my website here, along with both a horizontal and vertical version.

Monday, July 15, 2013

Complete site relaunch!

So... you may have noticed that things look a little different around here! I have officially relaunched my website with a whole new design and new url. The address for the new website is:

http://www.visuallanguagelab.com

...while the new url for the blog is...

http://www.thevisuallinguist.com

The old blog RSS feed should still be working as normal. Old links should serve as redirects (if everything is working right). However, please update any links you might have, pass around links to the new ones, and enjoy the new site!

Friday, July 12, 2013

Comic-Con 2013

If you're going to Comic-Con, be sure to come by my talk on Friday July 19th at 2pm, room 26AB. I'll be talking about some of my experiments on "Comics and the Brain." Here's the full listing:


Comics Arts Conference Session #8: Comics Theory
Scott Daniel Boras (Winona State University) examines the sensory experience of reading comics, what it means to be a collector, and other aspects of materiality that might be lost or left behind when comic books go digital. Neil Cohn (University of California, San Diego) summarize what the latest research from cognitive neuroscience tells us about what happens in your brain when you read comics. Kay K. Clopton (Ohio State University) analyzes how speech, sound, and emotive effects feed into the experience of reading American and Japanese women's comics.
Friday July 19, 2013 2:00pm - 3:00pm 



Also... full website relaunching coming soon in just a few days!

Friday, June 28, 2013

ComicCon and websites

I've unfortunately been very busy lately, so blogging has taken a hit unfortunately. However, a few notes about upcoming things...

First, after two years away, I'll be back speaking at Comic-Con International this year! My talk will be about "Comics and the Brain" and what neuroscience can tell us about how what goes on in people's brain's while they read comics. The final schedule hasn't been set yet, but it looks like my talk will be on Friday afternoon. Stay tuned for more details.

Also, in the next few weeks I'll be rolling out my new website. I will be officially changing the url for both the main website and the blog (the RSS feed should stay the same though). So, in anticipation of those changes, please update your links to:

Website: http://www.visuallanguagelab.com

Blog: http://www.thevisuallinguist.com 

Both of those addresses will, for now, bring you to the existing websites. However, in about two weeks, everything will launch again afresh on those sites while I slowly close things down on the existing web addresses. Looking forward to it!

Finally, if you happen to be going to the Cognitive Science Society Conference in Berlin this year, I'll be giving a talk as part of the Glushko Dissertation Prize Symposium on Saturday morning.

Wednesday, June 12, 2013

Two for No: theory in practice

I'm happy to be able to announce that my friends Alexander Danner and Tym Godek have a new weekly comic series called "Two for No." What makes it even more exciting for me is that the comics are all based around ideas from my theories!

Several years back, I wrote a paper about "visual language poetry" for ComixTalk where I proposed a "formal" type of visual language poetry that did not imitate the structures of verbal poetry, like when people create "comic haikus." Rather, my poetic form—and my proposal for how to make more poetic forms in visual language—drew directly from the properties of the structure of the the visual language of comics.

That poetic form was called a "Reducto" and it's been played with a bit previously by Grant Thomas, who also explored some of his own manipulations of it.

Now, Alex and Tym offer a whole series that is based on these ideas. They also go further than just using my form and have started making another visual language poetic form, the "Dissociation."

When I lived in Boston, I'd often wax theoretical with Alex and visited classes he taught on comic literature and creation. He frequently queried my thoughts on how my ideas might translate back around to the "practice" side of things, and got very excited when I recalled these ideas about poetry. I'm really glad he and the ever-talented Tym have been exploring these further, and I was glad to be able to see a sneak peak as they started on them.

In fact, Alex's pushing me about this stuff lead me to seriously consider adding a chapter in my upcoming book about visual language poetry. Alas though, I started running out of space and it didn't make the cut. Who knows, maybe Alex and I will one day get around to writing the "visual language theory in practice" book we've discussed in the past, with a whole chapter on this stuff?

For now though, go check out "Two for No," bookmark it, and visit every Monday when they post new comics!

Monday, June 10, 2013

11 years and pretty cover

I always seem to miss these things when they happen, but as of May 31st, my website has now been online for 11 years!

The amusing thing is that as another year goes by with my website, I am currently working hard to completely redesign and relaunch the site with the new url, www.visuallanguagelab.com. As I've mentioned before, now would be a good time to update your links if you haven't already. Full launch of the new site should be in a month or so.

In the meantime, production is still mounting on my new book. I expect to receive the proofs any day now. Even though it won't be released until December in the UK and January in the US, it has a cover!  Check it out:


Monday, May 20, 2013

Updates and such

Things have been a bit busy lately, so I've had less time to devote to blogging (suggested topics always welcomed, FYI). I've been hard at work on several revisions of papers for journals, as well as prepping several more papers for submissions. I'm also gearing up to start running a new brainwave study here at UCSD, so that's exciting to have underway.

I've also been hard at work on a major redesign/relaunch of this website. The site is getting a new look, new organization, and even a new address. If you'd like to plan ahead for the switch, the new url will be: www.visuallanguagelab.com (currently a redirect to the present site, which will then be flipped on relaunch).

Finally, I'm happy to report that my upcoming book, The Visual Language of Comics, has now entered the production stage! It will be fun to see the proofs in a few weeks. For now though, it's exciting to see that my publisher has now created a webpage promoting the book, including a growing list of endorsements. Looks like there's even a page on amazon for it. Let the countdown until it's release in December begin!

Thursday, May 02, 2013

CFP: Interdisciplinary approaches to visual narrative

For those that might be interested, one of my projects is trying to organize a book summarizing important research on visual narrative. This book will be a companion volume to my book out later this year, The Visual Language of Comics. If you may be interested in contributing, here's a Call For Papers for it...

CFP: Interdisciplinary approaches to visual narrative

While there have been a growing number of books on comics in recent years, very few have addressed aspects of structure, particularly from theoretical, cognitive, or experimental points of view and outside the realm of literary or sociocultural theory. I am working to organize a compilation of important papers on the understanding of sequential images. Most of the chapters will be either 1) summary papers that provide extensive bibliographies that can provide an overview to students and a resource to other researchers, or 2) reprints of significant research that remain under-recognized or hard-to-find.

This Call for Papers asks for proposals for papers of two types of chapters focused particularly on research outside of English, presented for an English speaking audience:

1. Chapters that summarize, in English, advances in comic theory from non-English speaking researchers. Such chapters should be large in scope with extensive reference sections.

2. Translations into English of significant non-English comic theory (structural, cognitive, experimental, etc.) from important papers or book chapters.

Topics or chapters outside this scope may be considered, though best to contact me directly with inquiries. (Of interest may be: review papers of other types, historical development of comic “symbology”, empirically grounded discussions of differences between comics cross-culturally, etc.). Importantly, papers should be relevant not only to scholars of comics, but also to linguists, cognitive scientists, and psychologists.

Contributor Guidelines

1. Abstracts of 400-500 words accepted. Papers of 5000-9000 words, including notes and bibliography, accepted. Please also include a short biographical statement.

2. All documents should be submitted as Word or Word-compatible files. PDFs are also acceptable.

3. Submission deadline: May 15, 2013. June 15, 2013

4. Materials should be sent to me via email (Neil Cohn)

Sunday, April 28, 2013

Comics, games, and bad science

I'm often very excited when I find new research on comics, especially when it's experimental. There is so little done that it's a treat to find something I didn't know about. Unfortunately, sometimes my excitement at new data turns sour when I see what was actually done...

I recently found this study (pdf) by Kinzer and colleagues that compares the comprehension and eye movements of readers for narratives in comics and video games. Their main goal is to help provide support for the use of comics and video games in educational contexts.

In this study, they presented sixth graders with either a video game version of a story or a comic created from the images of the video game. Overall, they find that participants understood the story in the video game version better than the comic version. They also found people spent more time engaged with the game than the comic.

I would take all of these findings with a grain of salt...

...because the stimuli appear to be extremely confounded because the comic versions of the story appear to be so badly created. Judging by the example in the text, the comic pages clearly were created by someone who had no real fluency in the visual language of comics. This is clear at a glance just by the example that they provide in the paper:



First off, the images make it extremely hard to tell what's going on. Second, almost all of the balloons are placed outside of their originating panels to the extent that they completely overlap in panels forward and backward in the text. I don't even need to know what's in the text to know this will be confusing to a reader. This is so "illegal" in the rules of page construction that it is almost painful.

If this is their example (what is probably the best example they have), I shudder to think what other pages in the experiment look like. Seriously, if I wanted to design an experiment that had "incomprehensible comics pages" as one type of stimuli, I'd use pages like these.

It's no wonder they found that their participants had poorer comprehension for the comic version—their stimuli are the equivalent of trying to test English comprehension while using broken English. It tells you next to nothing of interest.

There are two main points I'd like to make about this:

First, good experiments are hard to design, and having something worth saying must follow from having successfully designed an experiment that can give you good information. It pays to be critical as a creator and reader of scientific research (no matter what the topic).

Second, doing experiments using the visual language of comics is not trivial. Stimuli cannot be created by anyone, regardless of their fluency in comic creation. Just because you can throw together some images and words into panels on a page does not mean you've successfully created an example of "native" visual language. Believing otherwise does a disservice to yourself and to others who might read and cite your research.



Kinzer, C. K., Turkay, S., Hoffman, D. L., Gunbas, N., Chantes, P., Chaiwinij, A., & Dvorkin, T. (2012). Examining the Effects of Text and Images on Story Comprehension: An Eye-Tracking Study of Reading in a Video Game and Comic Book. In P. J. Dunston, S. K. Fullerton, C. C. Bates, K. Headley, & P. M. Stecker (Eds.), Literacy Research Association Yearbook 61 (pp. 259-275). LRA: Chicago, IL.