Wednesday, February 17, 2016

Mayan visual narratives in the BBC!

I'm very happy to say that David Robson over at the BBC has a new article out discussing Jesper Nielsen and Søren Wichmann's chapter in my new book, The Visual Narrative Reader.  Their chapter, and the BBC article, examine the structural properties of Mayan visual narratives found on the sides of pottery.

There are a lot of great things in their chapter that motivated me to invite them to be a part of the collection. Foremost, they nicely show that these Mayan systems share many properties with the "visual languages" used in comics and other cultures, ranging from the way they show sequences to the way they use text-image relationships and graphic signs like lines to show smells or speech.

In my conception of sequential image systems being like language, there is no one visual language, but rather there are many throughout the world. In addition, just as spoken languages change and die off over time, so do visual languages. The system used in the Mayan visual narratives thus reflects a “(Classic) Mayan Visual Language” tied to a particular time period and location. Similarly, we could identify historical visual languages from different time periods all over the world.

I’ll point out also that this is different than saying that Mayans used “comics.” This is not the case. “Comics” are the context in which we use some visual languages in contemporary society, and casting that label back in time is inappropriate. Rather, they have a visual language that is used in its own context tied to its era.

What makes the Mayan examples nicely illustrative is that they are an older, historical version of this that is preserved in the historical record. The visual language used in sand drawings (also discussed in two chapters of The Visual Narrative Reader.) disappears once it is drawn, because of the medium of sand, while the accompanying gesture/signs and speech disappear because they are spoken verbally. This means there is no historical record of them. But, the Mayan examples on pottery and ceramics are drawn and include writing, those artifacts can provide a window into past human behavior as a multimodal animal.

Finally, what I really liked about this article—beyond the the subject matter—was the way in which the subject matter was analyzed using systematic linguistic methods. I think this nicely shows how much of what has previously been discussed in “art history” can really be transported to the linguistic and cognitive sciences given the theory of visual language. If we’re talking about the structure of older drawing systems, then we’re not discussing “art” per se, but rather are discussing ancient visual languages and their structure. Further focus like this can contribute towards building a study of historical visual linguistics that can then analyze such material the same way as we think of any other type of linguistic system.

Monday, February 01, 2016

New Paper: The pieces fit

Magical things happen at conferences sometimes. Back at the Cognitive Neuroscience Society conference in 2014, I ran into my graduate school friend, Carl Hagmann, who mentioned he was doing interesting work on rapid visual processing, where people are asked to detect certain objects within an image sequence that changes at really fast speeds (like 13 milliseconds). He noticed that I was doing things with image sequences too and thought we should try this rapid pace with visual narratives (similar to this old paper I blogged about).

Lo and behold, it actually happened, and now our paper is published in the journal Acta Psychologia!

Our paper examines how quickly people process visual narrative sequences by showing participants the images from comics at either 1 second or half a second. In some sequences, we flipped the order that images appeared. In general, we found that "switches" of greater distances were recognized with better accuracy and those sequences were rated as less comprehensible. Also, switching panels between groupings of panels were recognized better than those within groups, again showing further evidence that visual narratives group information into constituents.

This was quite the fun project to work on, and it marks a milestone: It's the first "visual language" paper I've had published where I'm not the first author! Very happy about that, and there will be several more like it coming soon...

You can find the paper via direct link here (pdf) or on my downloadable papers page.


Recent research has shown that comprehension of visual narrative relies on the ordering and timing of sequential images. Here we tested if rapidly presented 6-image long visual sequences could be understood as coherent narratives. Half of the sequences were correctly ordered and half had two of the four internal panels switched. Participants reported whether the sequence was correctly ordered and rated its coherence. Accuracy in detecting a switch increased when panels were presented for 1 s rather than 0.5 s. Doubling the duration of the first panel did not affect results. When two switched panels were further apart, order was discriminated more accurately and coherence ratings were low, revealing that a strong local adjacency effect influenced order and coherence judgments. Switched panels at constituent boundaries or within constituents were most disruptive to order discrimination, indicating that the preservation of constituent structure is critical to visual narrative grammar.

Hagmann, Carl Erick, and Neil Cohn. 2016. "The pieces fit: Constituent structure and global coherence of visual narrative in RSVP." Acta Psychologica 164:157-164. doi: 10.1016/j.actapsy.2016.01.011.