Sunday, July 20, 2014

Rules of Emoji

A friend passed along this article recently which describes research by Tyler Schnoebelen that explores the rules people use when writing with emoji/emoticons. When people hear that I work on "visual language" they often think of things like this. However, Schnoebelen actually did study these things, scouring through roughly 500,000 tweets and analyzed the types of productions that people made.

For example, he noted that people tend to place emoji at the end of clauses and sentences rather than the beginning. This makes sense, because people won't want to divide up their syntactic clauses. Instead, they'd prefer to use emoji like punctuation to divide up those clauses.

There also seems to be at least some linear causality when people are trying to communicate in emoji alone. The article notes that a story told through emoji would have to go in sequence, like this:
But, rearranging the order creates a different story, possibly less comprehensible:
This type of ordering could possibly use the rules in my visual narrative grammar, but in fairly stripped down and choppy form. Scrambling the order of units though is a common technique we've used before in studying the structure of sequential images.

These are particularly interesting for me because this type of things is what people often mistake my research for. However, emoji are not actual "visual language" as is laid out by my theory. Rather, they are at the intersection of visual language with writing—a conversion of sound into graphics (a learned synesthesia). They can act like "visual gestures," supplementing or enhancing the expressions of the text. Or, they can be like a pidgin, a hybrid communicative resulting from the intersection of rules of different systems, often using combinations based on basic rules of meaning without a complex grammar.

Nevertheless, emoji are particularly interesting because they still can reveal various preferences that people have for ordering meaningful information that is likely shared across various domains. For example,  another thing discussed in the article is that people will generally show an emotional state first, then show the cause of it, as in the laughing-monkey-poo example above and this one, where sadness precedes the reason why (a broken heart):
These examples use a structure common across communicative systems, which is to express the Agent (the doer of an action) prior to the Action. In this case, the Agent is sad, and the sadness is caused by the action of a broken heart. Often, this pattern will have a Patient (a receiver of an action) as well, in a canonical Agent-Patient-Act order. This pattern occurs very frequently in both full languages (such as Subject-Object-Verb ordered grammars), and in communicative systems that lack a full grammar (such as when non-sign language speakers are asked to communicate with only gestures).

I think there could likely be some interesting work done using experiments that ask people to communicate with emoji alone. That might be a nice follow up to Schnoebelen's work. He's done the corpus analysis of looking at what people have done in naturalistic expression. An interesting next stage would be to test various rules in experiments.

So... are emoji "visual language"? Not really in any full blown sense. However, they can tell us about verbal and visual languages, and the rules of communicative systems more generally.


(Wayback machine note: A long time back I did this related post about emoticons and the brain)

Thursday, July 10, 2014

Germany workshops recap

I'm a few weeks late to post on it, but my workshops in Germany were wonderfully fun! I've spent the intervening week traveling throughout the East Coast, so I'm only now able to get to posting photos and whatnot from the trip.

It was a great time, and I greatly appreciated everyone's enthusiasm, energy, and curiosities. The workshops were especially fun since I really only planned an introductory slideshow, and then I took requests for whatever the attendees wanted to learn the remainder of the day.

My first stop was Freiburg where I had a packed house of students and faculty from lots of different fields. We focused a lot on narrative grammar and page layout (as we did later in Bremen), but also quite a bit on cross-cultural studies. Here's a group photo of those who survived the whole 8 hour workshop!


A few days later I made my way up to Bremen, where we had my second full day workshop. We again focused on narrative grammar and page layouts. But, I got to present some as-yet-unpublished theoretical work on multimodality (the paper for which I later ended up writing on the plane ride home). Alas, by the end of the 8 hours, I forgot to take a group picture, but it was quite the fun time. It also got covered by a German radio station, which you can listen to here.

I then gave my last talk as part of a series on film, which meant I had to discuss at least some ideas about how to apply my narrative grammar to film. That talk was in an amazing circular room at the University of Bremen that made me feel like I was an old Greek scholar or something (bring me a toga!). Also quite a fun experience, but only 2 hours so not nearly the marathon of the workshops.



All in all it was a fun and educational trip, definitely for me and hopefully for the people who came out to learn and work with me. So, special thanks to John Bateman, Janina Wildfeuer, and Stephan Packard for bringing me out there and for being such great and hospitable hosts, and thank you to everyone who came out to participate and listen to the talks!

If you couldn't come see me this trip to Germany, never fear! I'll be back again for a conference on "Empirical approaches to comics" in Berlin this September 19 - 20. I would of course love to do more workshops during that time, so contact me if you'd like to have me!

Monday, July 07, 2014

New article: The architecture of visual narrative comprehension

I'm happy to announce that I have a new article out in the journal Frontiers in Psychology titled "The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics."

This is a "Focused Review," so it summarizes the research that has been done on narrative structure, page layouts, and panel framing… and then tries to integrate them together! The intent is to show that each of these structures operates independently of each other, but that they all interface to create a larger whole than just the parts.


Abstract:

"How do people make sense of the sequential images in visual narratives like comics? A growing literature of recent research has suggested that this comprehension involves the interaction of multiple systems: The creation of meaning across sequential images relies on a “narrative grammar” that packages conceptual information into categorical roles organized in hierarchic constituents. These images are encapsulated into panels arranged in the layout of a physical page. Finally, how panels frame information can impact both the narrative structure and page layout. Altogether, these systems operate in parallel to construct the Gestalt whole of comprehension of this visual language found in comics."

Cohn, Neil. 2014. The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics. Frontiers in Psychology. 5:680. doi: 10.3389/fpsyg.2014.00680

Wednesday, June 11, 2014

Travel plans and Mailbag

I leave for my upcoming Germany trip on Monday! I'm very excited about the workshops and can't wait to jump into them. As it turns out, I've been invited to yet another conference in Germany held in Berlin this September, so I'll be headed back out there in a few months as well! I'll keep people posted as that develops.

I'm also very proud of how well my students did this quarter in my Cognition of Comics class at UCSD. My 24 students all did research projects on some aspect of visual language theory, be it running experiments or coding through lots of comics to see their properties. We ended the quarter with two weeks of presentations where they shared their work with the class. All the projects turned out great, but we'll now be working on transforming several exemplary ones into publications. Really looking forward to sharing this work with people.

And finally, I got an interesting question over on my Facebook page. I always enjoy answering people's questions, so I figured I should bring it over here too. Ryo Infinity sent me this message:

"Research by the University of College London asserts that anyone can learn how to draw - which contradicts some of your claims about improvement after the "critical stage". "In fact, say scientists, while some are born with natural talent, anyone can learn to draw well. In research presented at a recent symposium at Columbia University and soon to be published by Columbia University Press, Chamberlain and her colleagues found practicing drawing significantly improved people's abilities over time, as rated by other people who participated in the study. " What are your thoughts on this? The researchers opine that anyone can learn how to draw well after thousands of hours of practice - even if they start after puberty"

And here's my now slightly-expanded reply:

"Thanks for the question. I don't think of a "critical period" as a hard line that then prevents you to draw (it's not a switch that turns off). It's a diminishing ability to acquire drawing skills as you age. But, the decline seems to happen around puberty (just like in language acquisition). 
That said, I don't know of the work you're discussing, but my instinct is to say that yes, some people will naturally be better than others (just like some people naturally have a knack for languages) and practicing something over time will help improve it (just like languages). However, I'd want to know what the criteria are that they are using to judge proficiency. If it's just "other people in the study" then those aren't "fluent drawers" either—essentially like asking people who are learning Mandarin to judge the fluency of other learners. That makes no sense as a test of proficiency. I'd also likely challenge the basic foundations of what they believe drawing to be (and what they believe "good" drawing to be), since that's at the heart of the issue really."

Wednesday, May 21, 2014

Coming soon to Germany...

I'm very excited that my workshops in Germany are only a month away! I'll first be making a stop at Saarland University for a talk about the structure of events  on June 18th.Then the more extensive visits begin...

I'll be speaking on Saturday, June 21st at the University of Freiburg for an all day workshop. You can find more information here and a pdf poster here.

I'll then head over to the University of Bremen for a few events. First, I'll be doing another all day workshop on Monday June 23rd. As in Freiburg, I'll be discussing the basics of visual language theory, advice on how to go about doing research, and an extensive discussion of the structure and cognition of comics and visual narratives. I'm fighting the urge to over-plan too much, so that a good amount of the material can also be guided by the interests of people in attendance (if you know you'll be in attendance and want to influence the direction of discussion beforehand, let me know!).

Finally, I'll be speaking as part of the lecture series, "Recent Paradigms of Film Studies" on Wednesday June 25th. Here, I'll be talking about the structure and cognition of my theory of narrative grammar, and how it can apply to comics, films, and written discourse.

If you're in or nearby Germany, I hope to see you soon!

Wednesday, April 30, 2014

Visual Language research galore!

Can I brag for a moment? Not about me, really, but about my students!

This quarter I'm teaching a course on the "Cognition of Comics" for the UCSD Cognitive Science Department. While the classes do discuss experiments and studies that have been done related to how people understand comics, and how they differ across cultures, I wanted this class to give students the opportunity to do science rather than just learn about it.

So, the course actually revolves around students doing their own research projects related to topics in visual language. While I've tried to help guide them towards interesting projects, they've mostly come up with them on their own, and boy do they seem exciting!

We have 7 experiments that people are doing, investigating the cognition of "upfixes" (symbols that go above characters heads), how people use systematic representations in drawings, how people navigate page layouts, and even how color might affect emotion in comics.

We also have projects coding across many comics to see what the properties of various structures are. Before this class, there were less than ten studies doing this sort of corpus analysis on the structure of comics. There was a dissertation by Neff, McCloud's study of panel transitions, a few done by Charles Forceville, and two done by me. In this class alone, there are roughly 14 corpus studies!

The comparisons of populations range from looking at how various structural aspects of American Visual Language change over time in superhero comics, how structures might be similar or different between genres of Japanese manga (shoujo, shonen, sports, etc.), how structures might differ between comics done by English vs. French vs. Spanish speakers, and differences between American, Japanese, and/or Chinese comics. (among others)

These comparisons are looking at how panels frame information, the constraints on how manga use super-deformation, how "visual morphemes" and symbols are used across comics of different cultures, how schematic graphic information (like the way people draw eyes or hair) are systematically used across different authors, how page layouts are structured, and how text and images interact with each other. Plus, quite a few other topics!

Hopefully, many of these papers will find information that will be significant enough to publish, suddenly increasing what we know about the structure of different comics across the world by over 200%! It's going to be a very exciting quarter!

Tuesday, April 22, 2014

Post-talk and new Spanish article

Thanks to everyone who came out to hear my talk at WonderCon on Sunday! I got great questions and really enjoyed the discussion after the talk with people who hung around. The Con itself was pretty fun as well. As I commented with several people, it felt like ComicCon was back 20 years ago.

And, as you can see in the photo... I found someone walking around with an upfix! I think he was confused why I was so excited. Are you confused why I'm so excited? Then go check out my book, where discussion of "upfixes" (stuff floating above character's heads) receives a nice lengthy discussion!

In other news... the latest issue of Revista Exégesis is now out! This Spanish-language comics anthology has lots of good short stories, and, like all their issues, it features an article by me, translated into Spanish! This issue has a Spanish version of part of my Visual Language Manifesto, which discussed how visual language theory can inform a restructuring of the comic industry (originally written way back in 2004). Enjoy!

Wednesday, April 16, 2014

Wondercon and updates

Just a reminder that I'll be speaking at WonderCon in Anaheim this weekend! My talk on the Cognition  of Comics is going to be at 11:30 on Sunday. Hope to see you there!

Here's the description:

What happens in people's minds and brains when they read and create comics? Neil Cohn (University of California, San Diego) will present an overview of his new book, The Visual Language of Comics: An Introduction to the Structure and Cognition of Comics, which provides an extensive introduction to the cognitive science of comics comprehension. This discussion will cover the systematic components that make up unique and different panels, the grammar of sequential images and page layouts, cross-cultural differences in structure, and the newest neuroscience research on what the brain is doing while comprehending comics.
Sunday April 20, 2014 11:30am - 12:30pm 
Room 210BCD

Thursday, April 10, 2014

The Visual Language Fluency Index

One of the interesting findings throughout many of my experiments is that the comprehension of sequential images seems to be modulated by participants' "comic reading expertise." These effects are predicted by my theory of "visual language"...

If drawings and sequential images are indeed structured like language, then we should expect varying degrees of "fluency" across individuals based on their experience reading and drawing comics. Previous studies in Japan have supported this, finding that various aspects of comic understanding correlate with age and frequency of reading comics. Not only does this support my idea of "visual language," but it flies in the face of the assumptions that all (sequential) images are universally understood by everyone equally.

In order to study this type of "fluency," I created a measurement that calculates a number that can then be correlated with experimental results. In the first use of this metric, I found that brainwaves and reaction times correlated with people's fluency, and several studies since then have also found similar correlations. This study was predated in time (though not publication date) by my study of page layouts, which also found differences based on people's backgrounds, which was a precursor to the changes in the way I gathered this type of information.

I've now decided to name this metric the "Visual Language Fluency Index" (VLFI) and have decided to make resources available to anyone who might want to use it in their own experiments. Hopefully this can be helpful to anyone who is doing research or is planning to do research on sequential image comprehension.

You can now download a zip folder (direct link) from the Resources page of this site which contains a questionnaire for participants to fill out and an Excel spreadsheet to enter in this data, which will also calculate the VLFI scores. There is also a "read me" file providing documentation about the metric.

I'll make a final note as well that, although the VLFI score as it currently stands is very useful and has been proven to be a reliable predictor of comprehension in several studies, I'm not satisfied to leave it alone. Studies are already underway looking into how to improve the measurements and scale, which will hopefully make it even more reliable. Should anything change, I'll post about it here and update the files on the Resources page.

Thursday, March 27, 2014

Classes, speaking, and books... oh my!

I'm currently enjoying getting work done on my spring break, and have lots of fun news to report!

1. Classes!

First, I just completed grading the final exams for my very fun course on the "Language of Comics" for UCSD's Linguistics Department. Finals are always a chore to grade (even with my trusty TA), but they were made a lot more fun because they were filled with my theories and had awesome student drawn comics created to illustrate VL theory!

A class like this is pretty much one of a kind at this point, since I'm probably the only one teaching these sorts of classes. So, in order to help encourage and facilitate more classes like this, I've posted the syllabus on my "Resources" page. A direct pdf is available here. If you are looking to start teaching a class of this type, please feel free to contact me about listed readings that aren't published and/or for suggestions about homework assignments and exams.

Next week I start a new class on the "Cognition of Comics" for the UCSD Cognitive Science Department. This class is going to be a lot more research based, and will be based around students doing their own research project related to visual language theory. I'm very excited about it, and I'm looking forward to seeing all the great work they'll no doubt come up with.

2. Appearances!

While I'll be out in Germany to give several workshops in June, much sooner I'll be here in the States giving a talk at WonderCon in Anaheim in a few weeks. I likely won't be at Comic-Con this year due to a conflicting date, so this looks to be my big comic convention appearance of the year. My talk will be on Sunday morning of April 20th, at 11:30. I've got a whole hour where I'll be giving the basic overview of visual language theory and then answering lots of questions. I'll keep posting more info as the date gets closer, but come on out and see my presentation!

3. More books!

Finally, I'm excited to say I just received a contract for my next book! This one will be an edited volume that will act as a companion to The Visual Language of Comics, and can serve as a reader for future classes on visual language (I'll need to post an update to the syllabus when it's out!).

The book will bring together chapters from several world class researchers from various diverse fields who have all investigated some facet of visual narratives with regard to how they are structured, comprehended in cognition, or developed by children. Being able to integrate them into a cohesive volume will provide a great way to make these authors' work known to a broader community, and hopefully help sponsor the growth of this field.

Stay tuned for updates on its development, but I'm hoping for a release in late 2015.

Wednesday, March 19, 2014

June workshops in Germany!

I'm excited to announce that I have several upcoming appearances in Germany this June! It looks like I'll be having two main open workshops...

The first will be at the University of Freiburg throughout the day on Saturday, June 21st. I was invited to do this event by Stephan Packard, and it will be sponsored by the Institute for Media Culture Studies.

This workshop will be a compact and content filled introduction to my theories of visual language. I'll cover the basic ideas of this broad architecture, how it connects to other aspects of cognition, and how it can be used in practical analysis of comics and other phenomena. I'll be posting more information about this workshop in the coming weeks as we hammer out a few of the details and a website goes online for it. All are welcome to attend!

Finally, my most extensive stop will be at the University of Bremen, where I'll kick things off on June 23rd with another open workshop stretching throughout the whole day! (EnglishGerman) I was kindly invited here by John Bateman and Janina Wildfeuer for this event hosted by the Bremen Institute for Transmedial Textuality Research.

This workshop will again cover the basics of visual language theory, and will involve hands on discussion of how these principles operate throughout different comics and graphic communication.

As with my workshop in Freiburg, people are welcome to attend from all over. So, if you're in Germany, the Netherlands, Belgium or any other place nearby and want to learn about visual language theory, please come out for the workshop! Additional information about the workshop and registration can be found at this website.

Then, on June 25th I'll give an additional talk at the Bremen University lecture series on "Recent Paradigms of Film Studies" (EnglishGerman). This talk will provide an overview my theory of "narrative grammar," what the research on behavior and the brain can tell us about comprehending sequential images, and how these structures apply beyond static visual sequences like those in comics, but also to film, and verbal discourse.

I am very excited about these upcoming events, and to have the opportunity to share my work with so many diverse people. I'll be posting reminders and more information as the dates get closer, but if you're interested, please come to the workshops!

Friday, March 07, 2014

New article: Framing "I can't draw"

I'm happy to say that I have a new article (pdf) published in the journal Culture & Psychology! This one continues with my theories about how people learn how to draw.

In my previous article (pdf), I argued that drawings were structured like languages, and that learning how to draw involves learning a "visual vocabulary" from an external system. I also argued that the reason people feel that they "can't draw" is because they do not sufficiently have exposure and practice with these visual languages, and thus don't learn how to draw with "fluency" before the end of a critical learning period.

This new paper pushes this idea even further, and proposes that people's ability to draw is actually hurt by the way in which our culture thinks about drawings and graphic expression. As I've argued for a long time on this blog (with the tag "Art vs. Language") there is a perspective held about "Art" that pushes people towards drawing in unique and individualistic ways, admonishing imitation as a means of learning. This paper argues that the cultural set of assumptions including these ideas actually inhibits people's ability to learn how to draw.

A pdf of the paper is available here, while the official abstract and information from the publisher is here. Here's the abstract:

Why is it that many people feel that they “can’t draw”? In a recent article Cohn, 2012, I put forth a new theory that compared the cognitive structure of drawing to the cognitive structure of language. Like language, drawing uses schemas that combine in innumerable novel ways, and thus children learning to draw must acquire these schemas from the drawings in their environment. However, while most people in the United States and Europe “can’t draw,” Japanese children have far greater proficiency in drawing. This paper explores reasons for this cultural disparity in graphic fluency originating in the structure of the drawing systems in those respective cultures and the beliefs that frame ideas about drawing and art education. In particular, I explore the intriguing possibility that cultural assumptions admonishing imitation of other people’s drawings prohibits the acquisition of graphic schemas, thereby leading to people feeling that they “can’t draw.”


Cohn, Neil. 2014. Framing “I can’t draw”: The influence of cultural frames on the development of drawing. Culture & Psychology. 20(1): 102-117.