Monday, July 28, 2014

Recap: Cognitive Science 2014

This past weekend I got to enjoy the annual Cognitive Science Society conference in Qu├ębec City, Canada. Unfortunately, this year's conference conflicted with ComicCon, but in the end it was definitely worth heading up to Canada than just going a few miles down the road here in San Diego.

I had several presentations throughout the weekend, but the highlight was the "Cognition of Comics" symposium on the first morning. Over the past few years, I've started collaborating on visual language research with a few other researchers, and I assembled everyone together to talk about our work.

I offered both an opening overview of my/our line of research, and at the end of the symposium I presented a wides scale view of how cognitive science issues can be studied using comics. In-between, my collaborators presented. David Wagner (University of Stavanger) and Tom Foulsham (University of Essex) both presented the work they've been doing using eye-tracking to see where people look while reading comics. Then, John Drury (Stony Brook University) presented work of ours looking at brainwaves in the comprehension of visual narratives, math, sentences, and music. This work seeks to explore whether the same brain mechanisms are involved in processing all of these domains.

Beyond the incredibly interesting talks themselves, as far as I know, this symposium is the first time multiple empirical psychology studies about the cognition of comics has ever been put together. To me, it felt like the starting point of the nascent field of visual language research—very exciting!

This feeling was only reinforced because we had ongoing discussions about research and comprehension of visual language for basically four straight days. Looks like there's going to be some great research up ahead from all of us, and a lot of work to be done! Because of this, I imagine we might look back on this conference as a seminal point in the history of this research.

The other special thing about this conference was that my mentor, Ray Jackendoff, was awarded this year's Rumelhart Prize, which is the "Nobel" of the cognitive science field. It's hard to think of a scientist who is more deserving. The award not only honors his extensive scholarly contributions to areas across cognitive science (language, music, perception, consciousness, social cognition, etc, etc.), but also his teaching and mentoring. This is also quite deserved, as I can say as one of the few individuals to graduate with a PhD with him as my primary advisor.

As part of the award, Ray gave an impassioned talk about the need and benefit of "theory" in the cognitive sciences. I recorded video for the talk, and we're going to try to get it posted online for all to see. Link to come.

Sunday, July 20, 2014

Rules of Emoji

A friend passed along this article recently which describes research by Tyler Schnoebelen that explores the rules people use when writing with emoji/emoticons. When people hear that I work on "visual language" they often think of things like this. However, Schnoebelen actually did study these things, scouring through roughly 500,000 tweets and analyzed the types of productions that people made.

For example, he noted that people tend to place emoji at the end of clauses and sentences rather than the beginning. This makes sense, because people won't want to divide up their syntactic clauses. Instead, they'd prefer to use emoji like punctuation to divide up those clauses.

There also seems to be at least some linear causality when people are trying to communicate in emoji alone. The article notes that a story told through emoji would have to go in sequence, like this:
But, rearranging the order creates a different story, possibly less comprehensible:
This type of ordering could possibly use the rules in my visual narrative grammar, but in fairly stripped down and choppy form. Scrambling the order of units though is a common technique we've used before in studying the structure of sequential images.

These are particularly interesting for me because this type of things is what people often mistake my research for. However, emoji are not actual "visual language" as is laid out by my theory. Rather, they are at the intersection of visual language with writing—a conversion of sound into graphics (a learned synesthesia). They can act like "visual gestures," supplementing or enhancing the expressions of the text. Or, they can be like a pidgin, a hybrid communicative resulting from the intersection of rules of different systems, often using combinations based on basic rules of meaning without a complex grammar.

Nevertheless, emoji are particularly interesting because they still can reveal various preferences that people have for ordering meaningful information that is likely shared across various domains. For example,  another thing discussed in the article is that people will generally show an emotional state first, then show the cause of it, as in the laughing-monkey-poo example above and this one, where sadness precedes the reason why (a broken heart):
These examples use a structure common across communicative systems, which is to express the Agent (the doer of an action) prior to the Action. In this case, the Agent is sad, and the sadness is caused by the action of a broken heart. Often, this pattern will have a Patient (a receiver of an action) as well, in a canonical Agent-Patient-Act order. This pattern occurs very frequently in both full languages (such as Subject-Object-Verb ordered grammars), and in communicative systems that lack a full grammar (such as when non-sign language speakers are asked to communicate with only gestures).

I think there could likely be some interesting work done using experiments that ask people to communicate with emoji alone. That might be a nice follow up to Schnoebelen's work. He's done the corpus analysis of looking at what people have done in naturalistic expression. An interesting next stage would be to test various rules in experiments.

So... are emoji "visual language"? Not really in any full blown sense. However, they can tell us about verbal and visual languages, and the rules of communicative systems more generally.

(Wayback machine note: A long time back I did this related post about emoticons and the brain)

Thursday, July 10, 2014

Germany workshops recap

I'm a few weeks late to post on it, but my workshops in Germany were wonderfully fun! I've spent the intervening week traveling throughout the East Coast, so I'm only now able to get to posting photos and whatnot from the trip.

It was a great time, and I greatly appreciated everyone's enthusiasm, energy, and curiosities. The workshops were especially fun since I really only planned an introductory slideshow, and then I took requests for whatever the attendees wanted to learn the remainder of the day.

My first stop was Freiburg where I had a packed house of students and faculty from lots of different fields. We focused a lot on narrative grammar and page layout (as we did later in Bremen), but also quite a bit on cross-cultural studies. Here's a group photo of those who survived the whole 8 hour workshop!

A few days later I made my way up to Bremen, where we had my second full day workshop. We again focused on narrative grammar and page layouts. But, I got to present some as-yet-unpublished theoretical work on multimodality (the paper for which I later ended up writing on the plane ride home). Alas, by the end of the 8 hours, I forgot to take a group picture, but it was quite the fun time. It also got covered by a German radio station, which you can listen to here.

I then gave my last talk as part of a series on film, which meant I had to discuss at least some ideas about how to apply my narrative grammar to film. That talk was in an amazing circular room at the University of Bremen that made me feel like I was an old Greek scholar or something (bring me a toga!). Also quite a fun experience, but only 2 hours so not nearly the marathon of the workshops.

All in all it was a fun and educational trip, definitely for me and hopefully for the people who came out to learn and work with me. So, special thanks to John Bateman, Janina Wildfeuer, and Stephan Packard for bringing me out there and for being such great and hospitable hosts, and thank you to everyone who came out to participate and listen to the talks!

If you couldn't come see me this trip to Germany, never fear! I'll be back again for a conference on "Empirical approaches to comics" in Berlin this September 19 - 20. I would of course love to do more workshops during that time, so contact me if you'd like to have me!

Monday, July 07, 2014

New article: The architecture of visual narrative comprehension

I'm happy to announce that I have a new article out in the journal Frontiers in Psychology titled "The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics."

This is a "Focused Review," so it summarizes the research that has been done on narrative structure, page layouts, and panel framing… and then tries to integrate them together! The intent is to show that each of these structures operates independently of each other, but that they all interface to create a larger whole than just the parts.


"How do people make sense of the sequential images in visual narratives like comics? A growing literature of recent research has suggested that this comprehension involves the interaction of multiple systems: The creation of meaning across sequential images relies on a “narrative grammar” that packages conceptual information into categorical roles organized in hierarchic constituents. These images are encapsulated into panels arranged in the layout of a physical page. Finally, how panels frame information can impact both the narrative structure and page layout. Altogether, these systems operate in parallel to construct the Gestalt whole of comprehension of this visual language found in comics."

Cohn, Neil. 2014. The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics. Frontiers in Psychology. 5:680. doi: 10.3389/fpsyg.2014.00680