Thursday, April 10, 2014

The Visual Language Fluency Index

One of the interesting findings throughout many of my experiments is that the comprehension of sequential images seems to be modulated by participants' "comic reading expertise." These effects are predicted by my theory of "visual language"...

If drawings and sequential images are indeed structured like language, then we should expect varying degrees of "fluency" across individuals based on their experience reading and drawing comics. Previous studies in Japan have supported this, finding that various aspects of comic understanding correlate with age and frequency of reading comics. Not only does this support my idea of "visual language," but it flies in the face of the assumptions that all (sequential) images are universally understood by everyone equally.

In order to study this type of "fluency," I created a measurement that calculates a number that can then be correlated with experimental results. In the first use of this metric, I found that brainwaves and reaction times correlated with people's fluency, and several studies since then have also found similar correlations. This study was predated in time (though not publication date) by my study of page layouts, which also found differences based on people's backgrounds, which was a precursor to the changes in the way I gathered this type of information.

I've now decided to name this metric the "Visual Language Fluency Index" (VLFI) and have decided to make resources available to anyone who might want to use it in their own experiments. Hopefully this can be helpful to anyone who is doing research or is planning to do research on sequential image comprehension.

You can now download a zip folder (direct link) from the Resources page of this site which contains a questionnaire for participants to fill out and an Excel spreadsheet to enter in this data, which will also calculate the VLFI scores. There is also a "read me" file providing documentation about the metric.

I'll make a final note as well that, although the VLFI score as it currently stands is very useful and has been proven to be a reliable predictor of comprehension in several studies, I'm not satisfied to leave it alone. Studies are already underway looking into how to improve the measurements and scale, which will hopefully make it even more reliable. Should anything change, I'll post about it here and update the files on the Resources page.

Thursday, March 27, 2014

Classes, speaking, and books... oh my!

I'm currently enjoying getting work done on my spring break, and have lots of fun news to report!

1. Classes!

First, I just completed grading the final exams for my very fun course on the "Language of Comics" for UCSD's Linguistics Department. Finals are always a chore to grade (even with my trusty TA), but they were made a lot more fun because they were filled with my theories and had awesome student drawn comics created to illustrate VL theory!

A class like this is pretty much one of a kind at this point, since I'm probably the only one teaching these sorts of classes. So, in order to help encourage and facilitate more classes like this, I've posted the syllabus on my "Resources" page. A direct pdf is available here. If you are looking to start teaching a class of this type, please feel free to contact me about listed readings that aren't published and/or for suggestions about homework assignments and exams.

Next week I start a new class on the "Cognition of Comics" for the UCSD Cognitive Science Department. This class is going to be a lot more research based, and will be based around students doing their own research project related to visual language theory. I'm very excited about it, and I'm looking forward to seeing all the great work they'll no doubt come up with.

2. Appearances!

While I'll be out in Germany to give several workshops in June, much sooner I'll be here in the States giving a talk at WonderCon in Anaheim in a few weeks. I likely won't be at Comic-Con this year due to a conflicting date, so this looks to be my big comic convention appearance of the year. My talk will be on Sunday morning of April 20th, at 11:30. I've got a whole hour where I'll be giving the basic overview of visual language theory and then answering lots of questions. I'll keep posting more info as the date gets closer, but come on out and see my presentation!

3. More books!

Finally, I'm excited to say I just received a contract for my next book! This one will be an edited volume that will act as a companion to The Visual Language of Comics, and can serve as a reader for future classes on visual language (I'll need to post an update to the syllabus when it's out!).

The book will bring together chapters from several world class researchers from various diverse fields who have all investigated some facet of visual narratives with regard to how they are structured, comprehended in cognition, or developed by children. Being able to integrate them into a cohesive volume will provide a great way to make these authors' work known to a broader community, and hopefully help sponsor the growth of this field.

Stay tuned for updates on its development, but I'm hoping for a release in late 2015.

Wednesday, March 19, 2014

June workshops in Germany!

I'm excited to announce that I have several upcoming appearances in Germany this June! It looks like I'll be having two main open workshops...

The first will be at the University of Freiburg throughout the day on Saturday, June 21st. I was invited to do this event by Stephan Packard, and it will be sponsored by the Institute for Media Culture Studies.

This workshop will be a compact and content filled introduction to my theories of visual language. I'll cover the basic ideas of this broad architecture, how it connects to other aspects of cognition, and how it can be used in practical analysis of comics and other phenomena. I'll be posting more information about this workshop in the coming weeks as we hammer out a few of the details and a website goes online for it. All are welcome to attend!

Finally, my most extensive stop will be at the University of Bremen, where I'll kick things off on June 23rd with another open workshop stretching throughout the whole day! (EnglishGerman) I was kindly invited here by John Bateman and Janina Wildfeuer for this event hosted by the Bremen Institute for Transmedial Textuality Research.

This workshop will again cover the basics of visual language theory, and will involve hands on discussion of how these principles operate throughout different comics and graphic communication.

As with my workshop in Freiburg, people are welcome to attend from all over. So, if you're in Germany, the Netherlands, Belgium or any other place nearby and want to learn about visual language theory, please come out for the workshop! Additional information about the workshop and registration can be found at this website.

Then, on June 25th I'll give an additional talk at the Bremen University lecture series on "Recent Paradigms of Film Studies" (EnglishGerman). This talk will provide an overview my theory of "narrative grammar," what the research on behavior and the brain can tell us about comprehending sequential images, and how these structures apply beyond static visual sequences like those in comics, but also to film, and verbal discourse.

I am very excited about these upcoming events, and to have the opportunity to share my work with so many diverse people. I'll be posting reminders and more information as the dates get closer, but if you're interested, please come to the workshops!

Friday, March 07, 2014

New article: Framing "I can't draw"

I'm happy to say that I have a new article (pdf) published in the journal Culture & Psychology! This one continues with my theories about how people learn how to draw.

In my previous article (pdf), I argued that drawings were structured like languages, and that learning how to draw involves learning a "visual vocabulary" from an external system. I also argued that the reason people feel that they "can't draw" is because they do not sufficiently have exposure and practice with these visual languages, and thus don't learn how to draw with "fluency" before the end of a critical learning period.

This new paper pushes this idea even further, and proposes that people's ability to draw is actually hurt by the way in which our culture thinks about drawings and graphic expression. As I've argued for a long time on this blog (with the tag "Art vs. Language") there is a perspective held about "Art" that pushes people towards drawing in unique and individualistic ways, admonishing imitation as a means of learning. This paper argues that the cultural set of assumptions including these ideas actually inhibits people's ability to learn how to draw.

A pdf of the paper is available here, while the official abstract and information from the publisher is here. Here's the abstract:

Why is it that many people feel that they “can’t draw”? In a recent article Cohn, 2012, I put forth a new theory that compared the cognitive structure of drawing to the cognitive structure of language. Like language, drawing uses schemas that combine in innumerable novel ways, and thus children learning to draw must acquire these schemas from the drawings in their environment. However, while most people in the United States and Europe “can’t draw,” Japanese children have far greater proficiency in drawing. This paper explores reasons for this cultural disparity in graphic fluency originating in the structure of the drawing systems in those respective cultures and the beliefs that frame ideas about drawing and art education. In particular, I explore the intriguing possibility that cultural assumptions admonishing imitation of other people’s drawings prohibits the acquisition of graphic schemas, thereby leading to people feeling that they “can’t draw.”

Cohn, Neil. 2014. Framing “I can’t draw”: The influence of cultural frames on the development of drawing. Culture & Psychology. 20(1): 102-117.

Wednesday, February 26, 2014

Towards a visual sociolinguistics

I've heard some people complain that a "language" approach to drawings and sequential images is overly formal and washes out the ability to talk about socio-cultural issues. I actually think it's quite the opposite. Linguistics has long been acutely aware of social factors such as race, gender, geography, etc. which factor into the structure and usage of language, and are primarily studied under the subfield of "sociolinguistics." Visual language theory allows for us to open up a subfield of "visual sociolinguistics" as well, which I only point towards in my recent book but didn't have the space to elaborate.

Let's first look at the spoken language side of things. Sociolinguistics research has done well to show that speech marks people from particular locations, class, race, sex, gender, sexual orientation, and a host of other factors. You can tell these things by the sound of the speech ("accent"), the words they use, the way they interact in a discourse (When do you give feedback? When are you silent?), and many other facets of language structure.

(Note: cognitively, there is no difference between a "language" and a "dialect" or "accents." They all have the same cognitive status, and social factors are what lead to one type being elevated to being a "language" while degrading others as "dialects.") 

Additionally, stereotypes are created about people based on the patterned way in which they speak. For example, the "standard" American English dialect typically reflects the speech of people in media, and is used by educated, white, upper middle-class individuals, which enjoy a social status higher than others in American society. Other dialects of American English do not receive this designation as "standard" because they are used by lower-class, lower-educated, and/or minority individuals. Such stereotypes and social hierarchy is reflected in (and unfortunately often reinforced by) the languages that people use.

Similarly, visual languages carry social information as well. Let's start with a very obvious example, which is the use of highly charged and prejudicial graphic representations. For example, the depictions of minorities throughout the early part of the 20th century were extremely racist, including Africans, African-Americans, Asians, and others (let alone the demeaning behavior often taken by those characters). These highly loaded and prejudicial representations are the visual language equivalents of highly racist words, here tied to the specific visual languages used predominantly in America and Europe.

Both racist words and racist images reflect patterns stored in the heads of those who produce and comprehend them, and such representations carry with them prejudicial associations (whether intentional or not). This is why in many cases we strive to push such representations out of the use in our languages (both spoken and visual).

Now, I wouldn't say that all drawings are somehow intrinsically prejudicial, but I would say that just about all drawings carry with them sociocultural associations that mark them with information about their social context, the social group of their creator(s), and other socially loaded information in just the same way that spoken languages do.

As I've stressed, drawing systems are not universal, and there are many diverse visual languages in the world, each carrying their own social connotations about status and social groups. By seeing a drawing in a "manga style" (Japanese Visual Language), you recognize it as either coming from Japan, or coming from someone who associates with that visual language. If you are familiar with manga, you might even be able to tell the differences between the various "visual language dialects" that exist between Japanese visual languages, most often associated with genres (for example, the differences between the visual language dialects in shonen and shojo manga). If you're not familiar with manga, such diversity may not be readily apparent and it might "all look the same." This is the same as spoken languages: if you speak Japanese, the diverse varieties of spoken Japanese might be very clear, but if you don't speak Japanese, it may all sound the same. (And the same goes for English to non-English speakers, and every other language). 

What's more, just like with languages, people might judge a visual language based on the values they place on the visual language they most associate with. For example, people who exclusively like manga may find the drawing style (graphic structure) and storytelling (narrative structure) of American superhero comics to be hard to read and/or distasteful. The reverse may also be true. While these opinions may to some degree be chalked up to "taste," I believe they also reflect the social biases that come with familiarity to a particular visual language. You prefer the visual language of your "in group" while possibly dis-preferring the visual languages from outside that group.

This just touches on a small overview of what visual language theory can offer research and considerations of the social implications of drawings and visual representations. Far from the "cold and formal" view of this approach, I think visual language theory actually puts us in a better position to discuss these issues of diversity, prejudice (implicit and explicit), and social relations in drawings from a cognitive viewpoint.

Monday, February 17, 2014

Fantom Comics interview

For anyone who missed my recent video interview with the Fantom Comics store, you can now watch it all online here or at youtube.

We had some technically difficulties getting it up and running, so that limited some of the questions we got from the Internet audience. I also had a screensharing mishap, so about a third of my presentation is only audio without the accompanying slides. But, I've been told it was still entertaining, and I had a great time doing it. Quite glad the folks at Fantom Comics invited me to do it, and I'd love to come back and do another one.


Tuesday, February 11, 2014

Fantom Comics Live Video Podcast

I'm excited to say that this coming Sunday at 6:30pm Eastern Time I'll be doing a live video podcast in conjunction with the Fantom Comics store in Washington DC. You can watch online via a Google Air Link,

I'll be taking questions directly from the crowd in the store and from those posted on the Facebook event page which you can also sign up at. For those who are in DC and can make it into the store, they'll be selling discounted copies of my book that are signed and include unique custom doodles.

So, if you have any questions for me, would like to participate, or just want to watch, please tune in!

Here are the links one more time:

1) The Fantom Comics webpage post
2) The Facebook Event
3) The Google Air Link where the show will be

Tuesday, February 04, 2014

Commander Mark and visual vocabularies

I got a blast from the past today when my old friend and awesome comic artist Dan Fraga posted this video of "Commander" Mark Kistler's classic drawing show from PBS, The Secret City:

I took Commander Mark's drawing classes during the summers when I was a kid. He was actually from my hometown, and his mother and brother also taught at my high school. I think a childhood friend of mine even sang in the theme song for his show! Apparently his more recent work has even earned him an Emmy, which is most certainly deserved.

Those lessons I'm sure made an early impression on my thinking about how drawing is learned and understood. His basic ideas certainly align with my own: drawing is made up of a "graphic vocabulary" that is used and manipulated to create novel whole drawings. In fact, I still draw various things exactly as I was taught by Commander Mark when I was nine or ten years old.

I also frequently use one of his tricks during lectures and presentations. Just like he did (still does?), I'll ask the crowd to draw something simple, like a house and a flower, and I give them only 10 seconds to do it. Inevitably, they will all draw something exactly the same (which would look exactly like what you're thinking now), and I'll have those images appear on the screen. It's taken as amazing, as if I had some mind reading powers.

Really though, this exercise nicely shows how we all share common cultural schemas in drawing, despite most people not progressing in their graphic fluency. These schemas are analogous to conventional gestural "emblems" like thumbs up or "OK." These gestures might be systematic and shared across our culture, but most people still haven't reached the ability to use a full sign language (which has its own vocabulary and grammar). Similarly, these simple graphic schemas are patterned forms of graphic expression that people are able to produce, even if they haven't fully acquired a complete visual language. I discuss these ideas more in my book and paper "Explaining 'I can't draw'."

In any case, if you're looking for works "to recommend for kids learning to draw," Mark Kistler's books are at the top of the list.

Wednesday, January 29, 2014

Some January notes...

Here's just a few notes from recent developments...

For all you Spanish speakers, a new translated article of mine appears in the latest Revista Exégesis, revisiting some of the ideas I had about the comic industry and definitions of "comics" from my "Visual Language Manifesto."

I recently submitted a response to an article in Trends in Cognitive Science that discussed the relationship of syntax and action. If you're interested, you can read it here. That page also has a nice response written by David Kemmerer, who motivated me to submit mine.

For those of you in Germany, it looks like I'll be traveling there to give some workshops throughout the summer. I'm very excited about these, and will be posting more about them once they develop further. I currently have a few stops scheduled, but if you're in Europe and might be interested in adding another event on my calendar, shoot me an email!

Finally, though official US release date for The Visual Language of Comics is still not even until tomorrow (1/30), it's already had to go for a second printing. So, thanks for all who have gotten it so far!

Wednesday, January 22, 2014

VLOC released in the US!

I'm excited to say that it looks like my new book, The Visual Language of Comics, is now available in America!

The official release date for the book is still set for next week (January 30th), but it appears that you are now able to order (and hopefully receive!) copies ordered online.

So, if you've been patiently waiting... the wait is over!

Since my post related to the book's release in Europe discussed its contents, maybe here I'll mention a bit about its intent.

This book is essentially an overview of the ideas I've been developing over the past 15 years, and is an invitation to both engage with the ideas and (where appropriate) participate in the discussion and/or research.

Most of the chapters here are simplified or summarized versions of material that appears throughout my research papers. Others provide new theories that I have kicked around for several years but never articulated in a broader discussion. The chapter on "morphology" in the "visual language lexicon" is one of these, and was one of my favorite chapters to write (and draw!).

In all cases though, this work remains an introduction. Ten years ago in 2003, when I self-published my first book on this topic, an advisor of mine remarked that he found it amusing that I titled the book *Early* writings on visual language, because it forecasted that there would be subsequent writings that would supersede its ideas (which was true, even at the time of printing the book).

That same sentiment holds true for this book. In the title, it says "Introduction" for a reason. This is the first presentation to a broad audience of a research program that I intend to develop further over the rest of my career (and hopefully other people's careers too). In none of the chapters do I spell out the entire theory of that topic—I could expand all of them to triple the length. (Note that my constraint didn't stop at individual chapters. Due to a word limit on the book I also scrapped several other whole potential chapters)

I also do not consider these statements to be the final word—no scientific writing ever should. Rather, this book aligns the questions in the ways I think they should be asked, and provides a preliminary analysis of these structures, often just to illustrate how such analyses should be undertaken. Asking these questions allows us to recognize that there are many more questions than we have answers for at present.

This book is truly is an introduction meant to inspire and invite. I hope you take up the chance to join in—now we can really get things started.

Thursday, January 16, 2014

Metaphors in the Bayeux Tapestry

Here's an interesting article examining semantic aspects of the Bayeux Tapestry that just appeared in the latest issue of Review of Cognitive Linguistics.

I've written before about Charles Forceville's work applying conceptual metaphor theory to aspects of the visual language of comics. His approach has subsequently been applied by other researchers looking at comics, animation, and film. Now, here's an article that examines conceptual metaphor in historical visual language, particularly the images in the Bayeux Tapestry.

There are several things that are interesting in and about this article. First off, it is one of the few explicit studies of historical examples of visual language that have been done using a linguistic analysis (another being this).

Second, the article nice details various systematic representations in the "drawing style" of the Bayeux Tapestry. As I argue in my book, drawing systems use a "visual vocabulary" of graphic patterns that are reused when people draw. The "styles" of different cultures' drawings emerge from various people sharing the same visual vocabulary. Here, we see the visual vocabulary of medieval English drawing detailed in systematic ways—perhaps "Medieval British Visual Language"?

Finally, the actual metaphors that are examined relate to various aspects of conveying emotion. Conceptual metaphor in general relates to how one domain maps to the ideas in another domain. For example, the metaphor LIFE IS A JOURNEY, is invoked when saying things like she's got a long road ahead of her or he's starting on a path of discovery. In these cases, aspects of living life are mapped to concepts of travel or a journey.

The metaphors described by this article mostly involve aspects of emotion, such as ANGER IS HOT FLUID IN A CONTAINER, which in comics and cartoons occurs when steam comes out of someone's ears. Similar metaphors are described here, depicted in the Bayeux Tapestry.

Of particular  interest though, is that the authors claim that the same metaphors appear here in the tapestry as in the Old English language. This taps into issues related to how much connection there is between the conceptualization in spoken and visual languages, and in questions of cross-cultural universality and diversity.

As always, I'd love to see more careful, systematic research of this sort with existing comics and historical representations alike.

Full abstract:
Following Forceville (2005, 2011), in this paper I show that the same conceptual models underlie the expression of Old English emotions in both the language and the visual modes. Kövecses (2000, 2005) and Stefanowitsch (2004, 2006) have shown that verbal expressions and idioms used to describe emotions can be traced back to a limited number of conceptual metaphors. In the light of these findings, I will analyze here the pictorial representations of emotions in the Bayeux Tapestry, an 11th century embroidered cloth that narrates and depicts the events that led up to the Norman Conquest of England and the invasion itself. The tapestry, which has been described as an example of early narrative art (McCloud, 1993, pp. 12–14), shows hundreds of human figures in an astounding range of poses and circumstances. My analysis of the set of pictorial signals used in the Anglo-Norman Bayeux Tapestry to represent emotion types such as ‘anger’, ‘grief’ and ‘fear’ shows that (1) Anglo-Norman artists used a well-organized set of visual stimuli to convey emotion-related meanings in a patterned way, that (2) the same idealised conceptual models are shared by verbal and visual modalities, and that (3) whereas verbal expressions of emotions regularly draw on non-embodied, behavioural concepts, visual representations show a clear preference for embodied container concepts.

ResearchBlogging.orgE. Díaz Vera, Javier (2013). Woven emotions: Visual representations of emotions in medieval English textiles Review of Cognitive Linguistics, 16, 269-284 DOI: 10.1075/rcl.11.2.04dia

Wednesday, January 08, 2014

Universality and diversity in visual languages

One of the interesting byproducts of my theory of visual language is the implication that drawings are not universal. Because images look like what they represent (the are "iconic), they are often taken to be universal in their ability to convey meaning. While this is often the case, it is not the whole story.

Visual Language Theory is committed to a cognitive viewpoint for looking at images. As I argue in my book and my article "Explaining 'I can't draw'," drawings are not created out of our perception of the world. Rather, we use a vocabulary of graphic "schemas" that then correspond in meaning to the way things look. For example, Jack Kirby had a particular way of drawing open hands that he used each time one was called for. This graphic pattern was stored in his mind, and connected to the concept of "open hand."

This means that in order to draw, each individual needs to build up a vocabulary of patterned ways to draw things. I've argued that this is one of the reasons why people end up not being able to draw proficiently: they don't build up (i.e. learn) a sufficient visual vocabulary of graphic patterns in this way, in the same way that learning a language requires you to learn the vocabulary of words (i.e. sound patterns connected to meanings).

This perspective flips the consideration of images away from perception (seeing images) to production (producing images). In order to view images, they first must be produced from the mind of an individual, and from that individual's cognitive patterns (*modern computers notwithstanding).

Ok, with that on the table, let's now talk about issues of diversity and universality.

In terms of comprehension, drawings are indeed fairly universal in their understandability. At least, most parts of them. The studies I've read seem to suggest that the iconic aspects of understanding images are fairly understandable across the world. People can generally understand basic images if they look like what they represent, such as a horse is understood as a horse, a car as a car, etc. (but not always stick figures, interestingly enough). This applies at least to basic line drawings.

The aspects of drawings that pose a challenge to certain people (such as those removed from Western society, like from African tribes) are the more "realistic" aspects of images. Things like depth (especially perspective) and realistic shading end up being less understood (compared to depth using size or simple drop shadows, which don't pose problems). There are some other variables too, but these are the basics. (Of course, highly conventional things like thought bubbles, speech balloons, and motion lines are also culturally relative, but we'll keep the discussion to the iconic aspects of drawings).

So, in comprehension, there is a balance of what appears to be universally understandable (i.e., recoverable from vision alone) and what relies on knowing aspects of culturally relative drawing drawing systems.

In production, the diversity should be even more apparent. Because drawings come from the minds of individuals, a person's "drawing style" reflects the visual vocabulary they have developed. If various people share common patterns, we can say that they all draw in a common visual language. This gives rise to different visual languages in the world, just like there are different verbal languages spoken by different populations of people with common patterns in their heads.

Thus, why do American comics and Japanese manga look different? It's because they are produced using different visual languages.

In this way, even though drawings may be iconic—resembling what they look like—they are not universal. Like any other language, visual language remains culturally relative and diverse.

Having made this distinction, the questions then become 1) what underlying aspects of drawings are indeed universal, and 2) what are the ways in which different systems diverge. Answering these questions requires as serious and expansive research as would be necessary to describe the "universality" in any other linguistic system.