Tuesday, December 31, 2013

Updates and such

With the year closing, now's as good a time as any to survey some of the things going on around here…

Now that The Visual Language of Comics is out in the UK/Europe, I am eagerly awaiting its release in the US at the end of January. For those of you who are reading it, I hope you're enjoying it, and I encourage you to post a review of it on its Amazon page. I am of course always welcoming of feedback directly or posted here in the blog comments as well.

For those of you who speak German, the article about my work in the magazine Der Spiegel has now appeared online in full.

For those of you who speak Spanish, my articles have continued to be translated each month in the online magazine Revista Exégisis. The latest issue translated my article on "natural" visual language poetry (English article), and also includes example strips from my friends Alexander Danner and Tym Godek's Two for No.

This coming year looks to be an exciting one. Beyond the full book release, I continue to work on a few exciting studies here at UC San Diego that are currently underway. I should have several papers appearing in journals within the coming months, and have plans to submit several more soon. Among these projects is actually my next book, which will be an edited volume combining the work of several researchers who have contributed to this broader field that I'm working to establish (most all of whom are cited in The Visual Language of Comics). More on this as it develops...

Finally, I am very much looking forward to this coming school year here at UC San Diego. I'll be teaching a "Language of Comics" class for the linguistics department in Winter Quarter, which will focus on the structural aspects of visual language. This class builds off of the one I taught several years ago at Tufts, but now with the added bonus of having a coherent textbook in my book! Then, in Spring Quarter we'll be moving on to a class on the "Cognition of Comics" for the cognitive science department. There the emphasis will be on the experimental and corpus research analyzing visual languages, and students will be guided through doing their own research projects on visual language. I'm really excited about this class!

Here's looking forward to 2014!

Monday, December 23, 2013

Der Spiegel article

For all you German speakers, the latest issue of Der Spiegel has an article about my research. Unfortunately, the full content is not available unless you're a subscriber, but a friend of mine has translated the preview content. Here's what it reads in English…

"An American psychologist claims that pictures speak their own language, with the pictures following a fixed grammar. Does the brain perceive 'Peanuts' in the same manner as Goethe's 'Faust'?
On the day that the pictures learned to speak, Neil Cohn didn't understand a word. In front of him lay a pile of manga comics that a friend had loaned him. He flipped through the stacks, and was bowled over by what he saw. Neil didn't speak Japanese, but he read. Picture by picture, he deciphered the story. Hours later his head hurt.

Years later Cohn worked as a comics illustrator, and now as a psychologist, and he sometimes wonders whether that day and the Mangas had something to do with what would become of him. That day, he now believes, he learned a new language; a language that has nothing to do with words."

The article is actually much longer (2 magazine pages), so if you read Der Spiegel, go check it out!

Thursday, December 19, 2013

Syntax, narrative, grammar… what!?

One of the aspects of my theories that people often find confusing is my claim that sequential images use a "narrative grammar" that is analogous to the grammar in sentences. This is confusing to many, because clearly individual images hold much more information than individual words.

This is true, but the analogy I make cuts beyond merely how much information is being conveyed, and involves deeper aspects of cognition related to their function, structure, and processing.

At the level of information, yes, panels contain much more information than words: they contain information more analogous to whole sentences. In fact, I would argue that the "narrative structure" that guides sequential images is the same structure that guides discourse and narrative of verbal stories. My theory works equally well for both of them.

So, why is there an analogy between narrative and syntax then?

First, though sequential images convey more information than individual words, narrative structure serves the same function as syntax to the communicative system. 

Both syntax and narrative function to package meaning in ways that are linear and coherent. Syntax itself is not meaning. The sentence Paul kissed Samantha is the same in meaning as Samantha was kissed by Paul, though they have different syntactic structures. Similarly, the same meaning can be conveyed in multiple ways in a sequence of images. These are the "storytelling" choices that authors make. "Narrative grammar" is the system that describes these rules.

Second, these rules governing narrative are structured in similar ways to syntax. 

There are two main aspects of this analogy. Both syntax and narrative use "grammatical categories" that provide functional roles to their units. In syntax, these categories are nouns and verbs. However, the categories for sequential images are not nouns and verbs, but rather things that I call Establishers, Initials, Peaks, Releases, etc. These categories are determined both through aspects of their meaning, but also through the ways in which they appear throughout a sequence—their "distributional trends."

Also similar to syntax, narrative is organize into hierarchic groupings. Just as syntax cannot simply move from "transitions" between one word and the next, sequential images are not understood by linear relationships between juxtaposed images. You need to be able to create groupings of images, and connect panels together across non-juxtaposed distances (such as in the above example excerpted from Tym Godek's One Night). In both syntax and narrative, these groupings then create several other constraints related to resolving ambiguity, making distance connections, coordinating the order of segments, etc.

Thus, the architecture of the "grammars" of both syntax and narrative are built in similar ways.

Third, the processing of narrative structure appears to be similar to the processing of syntax in the brain.

While this research is new and ongoing, the results so far from experiments on the comprehension of sequential images suggest that the same brain responses occur for the processing of narrative grammar as syntax (see here and here). In my experiments, I have tried to replicate the methods of classic experiments on sentence structure, and I have found similar results as these studies (short versions: here and here).

Now, does the fact that the same brainwave effects occur to syntax and narrative mean that the "same grammar" is being used in both? No. This does not mean that sequential images use the syntax of nouns and verbs. Rather, it means that the brain is treating both of these systems—these grammars—in similar ways.

For more information about my theories of Visual Narrative Grammar, check out my new book, The Visual Language of Comics, and in downloadable papers on my website.

Wednesday, December 18, 2013

Fast Co.Design article

Fast Company magazine's Co.Design site has a write up of my research over at their site.

The article focuses mostly on my "(Pea)nuts and bolts of visual narrative" article from last year (downloadable here). That was the first of my studies looking at the brain to understand the narrative grammar of sequential images, so it was fairly significant for me. Hopefully some of my more recent brain studies should be coming out over the next year!

Thursday, December 05, 2013

VLOC released in Europe!

The day has finally come…My new book The Visual Language of Comics is now available for sale in Europe! My publisher (Bloomsbury) has an interview with me over at their blog to mark the occasion.

The US release is still a little ways to go (January 30), so across the intervening time I plan to do several posts discussing some of the book. So, let's start with a little overview…

The book is divided into two main sections. Part 1 discusses "Structure and cognition" while Part 2 discusses "Visual language and the world."

The first part of the book lays out the basic structures involved in visual language:

What are the structures of drawings? Is there a lexicon of images? How do sequential images form coherent narrative sequences? What are the rules for navigating page layouts?

This first half also has a chapter summarizing the various research that has been done on cognition and comprehension. I believe that any theory about the understanding of comics should be able to be backed up by psychology experiments, both on behavior and the brain. So, this chapter summarizes this work done by me and others.

The second half of the book spends three chapters exploring how visual languages are different across the world. Visual language is not universal, and just like spoken and signed languages are diverse across the world, so are visual languages.

Why do American and Japanese comics differ in how they look? It's because they are written in two different visual languages: American VL and Japanese VL. So, there is a chapter on each of these systems, using them as platforms to discuss a variety of other issues involved in the learning and diversity of visual languages.

The third chapter in this section then provides an overview of Australian Aboriginal sand drawings—a visual language far removed from the context of comics, yet still a visual language in the sense of my theory. This chapter is especially important for clarifying how visual language is not just about comics, and it also provides a nice contrast to the systems found in comics. If we are to get at what might be "universal" about visual languages and drawing systems, then its important to look at these types of comparisons in detail.

That's the overall layout of the book… in future posts I'll try to discuss other "behind the scenes" aspects of the book's content, intentions, and preparation.