Tuesday, October 14, 2014

Using the theory of narrative grammar

As people have now started reading my book and papers, they've naturally started to try to apply my theories of "narrative grammar" to sequential images found in comics. My "narrative grammar" is a model of how the "storytelling" of sequential images is understood, which extends beyond previous approaches like Scott McCloud's theory of panel-to-panel transitions.

As people have now opened up comics, and tried to use the theory to describe randomly found pages and sequences, they have no doubt discovered that it is not easy. In fact, they may have thrown up their hands in frustration. If this theory is psychologically real then, why is it so hard to analyze sequences? Does this mean the theory is wrong?

No. There are many reasons why analysis may be challenging...

1. Multimodality

First, the theory is designed to account for sequences without text. Once text is introduced, the sequence must balance both structure (i.e., grammar) and meaning in multiple modalities, and it becomes manifestly more complex. I'm hoping to have a paper detailing this out soon.

2. Procedures

Ok, so what about wordless sequences? Just as it would be really difficult to just read a paper about linguistic syntax and analyze sentences, this theory requires some training to do it properly. At the very least, it helps to follow procedures for how to go about analysis.

Even I don't just look at a sequence and immediately know what the analysis is. I go through a series of procedures that tests the structure at each step of the way (these procedures are found in both my book, in Chapter 6, and the section on "diagnostics" in the "Visual Narrative Structure" paper, though not enumerated for how to go through them).

Here's how I train students in my classes and workshops to analyze sequences: the first thing we do is find the Peak panels. The rest of the sequence hangs around the Peaks, so it's the first thing we find. How do we know what is a Peak then? We test panels by trying to delete them (if the sequence is weird without them, then it's likely a Peak) or replacing them with an action star (if it does replace, it's likely a Peak), or deleting everything else except them (Peaks should be able to paraphrase a sequence on their own). From here, other procedures are then used to determine the other categories and the hierarchy of the sequence.

The point being: you can't just look at a sequence and intuit the structure (even me). That's why I describe tests and diagnostics, so that you can do it without just relying on intuition at every step of the way. Procedure matters.

3. Theory as framework

Third, the theory is a framework, not a catch-all. Theories of syntax in language are not "fully formed" when they are written about, and no theory of syntax in any book or any paper—of any linguistic model—is designed to immediately encompass every sequence one could encounter "out of the box." Rather, the theory provides a framework by which to account for the various diversity found in sentences. One then uses the framework (or changes the framework) to describe the various phenomena that are found in actual language use.

I consider a theory to be "good" if it can do two things: 1) account for more phenomena that is found in a structure (here, visual sequences) than other theories, and 2) can be revealed in experimentation to have psychological validity.

Much of syntactic theory about language is not simply finding things in sentences and then describing them using a particular theoretical model. Rather, the examples found in sentence structures are both described with theories of syntax, but also are used to illustrate how they pose challenges to theories of syntax such that those theories must grow and change. Throughout the 1960s and 1970s, much of the "wars" that were fought throughout linguistics had this characteristic—finding various patterns of syntax that would force changes towards preferring one theory of grammar or another.

That said, my theory of "Visual Narrative Grammar" is not meant as an "out of the box" analysis tool that should apply to every sequence of images from a comic based only on the chapter from my book. There are many, many more traits of the theory that have yet to be published, all of which deal with more complicated sequences and the non-trivial issue of combining sequential images with text. What is in my book and papers so far is less than what I even teach in an introductory class on visual language: I have a draft manuscript of over 300 pages (and growing) detailing various phenomena with the theory, most of which hasn't been published yet.

In addition, various sequences should challenge the theory, which is exactly the method I've used for the past 15 years to build the theory in the first place. I've had a theory, then found sequences that force changes to the architecture, and then altered the theory to be able to account for those issues. It's an organic process. The theory gives us a way to discuss and analyze such complexity and see how it might work. It's a framework, not a catch-all, just like all linguistic theories.

This is exactly the opposite of something like panel-to-panel transitions which are based solely on the low-level changes in meaning that occur between images. Such a theory is simple—there will always be meaningful changes between panels, and so it always seems to work. That's its appeal. The problem is that such an approach doesn't explain much of the data, which is far more complex than such a simple approach can manage. Indeed, my approach first started by expanding McCloud's panel transitions, and then altering it as I found sequences that it couldn't handle.

The fact is, the way we express meaning—be it through verbal language, visual language, or their combination—is very complex. There are no simple answers, and we should distrust the simple answers that might be offered. Recognizing this complexity, and building a framework that can let us study it, is the first step to exploring how it is understood.

Thursday, October 02, 2014

New paper: The grammar of visual narratives

I'm very excited to announce that I have a new paper out in the latest issue of Neuropsychologia on how the brain understands sequences of images, "The grammar of visual narratives: Neural evidence for constituent structure in visual narrative comprehension." This was actually one of my dissertation projects, so I'm very happy to finally have it published.

In this paper, I provide evidence that sequential images are processed by creating "constituents"— groupings of panels—rather than just "transitions" between the meanings of panels. Furthermore, disrupting these groupings in the "narrative grammar" of sequential images evokes the same brain responses as when you violate the grammar of sentences.

You can download the full paper here (direct link to pdf). I've also made a video summarizing the paper:



Official abstract:

Constituent structure has long been established as a central feature of human language. Analogous to how syntax organizes words in sentences, a narrative grammar organizes sequential images into hierarchic constituents. Here we show that the brain draws upon this constituent structure to comprehend wordless visual narratives. We recorded neural responses as participants viewed sequences of visual images (comics strips) in which blank images either disrupted individual narrative constituents or fell at natural constituent boundaries. A disruption of either the first or the second narrative constituent produced a left-lateralized anterior negativity effect between 500 and 700 ms. Disruption of the second constituent also elicited a posteriorly-distributed positivity (P600) effect. These neural responses are similar to those associated with structural violations in language and music. These findings provide evidence that comprehenders use a narrative structure to comprehend visual sequences and that the brain engages similar neurocognitive mechanisms to build structure across multiple domains.



Cohn, Neil., Jackendoff, Ray., Holcomb, Phillip. J., & Kuperberg, Gina. R. (2014). The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension. Neuropsychologia, 64, 63-70. doi: 10.1016/j.neuropsychologia.2014.09.018

Thursday, September 11, 2014

New Paper: You're a good structure, Charlie Brown

Wow, big week for new papers! I've got another one officially out now in the latest issue of Cognitive Science, "You're a Good Structure, Charlie Brown: The Distribution of Narrative Categories in Comic Strips" (pdf). This one actually reports on the first studies I did on comics during grad school, way back in 2007! (Sometimes science takes awhile to get out...).

This study examines whether the narrative categories I've proposed for sequential images use patterned roles in sequences. There are four experiments we used:

Experiment 1: People took 4 unordered panels and ordered them into a sequence
Experiment 2: We measured how long people viewed panels in sequences where two panels were reversed in order
Experiment 3: People took 4 unordered panels, ordered 3 into a sequence and deleted 1
Experiment 4: People viewed 3 panel sequences where one panel had been deleted, and guessed where it was deleted from

Across all tasks, we found complementary results for how different types of panels behaved, showing that there are certain "distributional trends" for the roles that panels play in a sequence. For example, panels that were freely chosen to be deleted were not recognized when they were missing, but panels that were not often deleted were noticed when gone.

Plus, some panels can play multiple roles in a sequence, but not all panels have this ability. This confirms that some types of panels are flexible in the role they play in a sequence, but not all panels can go in any location in a sequence. This goes against the idea that "any panel can go in any position" in a sequence and still be meaningful.

Here's the full abstract:

Cohn's (2013) theory of “Visual Narrative Grammar” argues that sequential images take on categorical roles in a narrative structure, which organizes them into hierarchic constituents analogous to the organization of syntactic categories in sentences. This theory proposes that narrative categories, like syntactic categories, can be identified through diagnostic tests that reveal tendencies for their distribution throughout a sequence. This paper describes four experiments testing these diagnostics to provide support for the validity of these narrative categories. In Experiment 1, participants reconstructed unordered panels of a comic strip into an order that makes sense. Experiment 2 measured viewing times to panels in sequences where the order of panels was reversed. In Experiment 3, participants again reconstructed strips but also deleted a panel from the sequence. Finally, in Experiment 4 participants identified where a panel had been deleted from a comic strip and rated that strip's coherence. Overall, categories had consistent distributional tendencies within experiments and complementary tendencies across experiments. These results point toward an interaction between categorical roles and a global narrative structure.


Full Reference:

Cohn, Neil. (2014). You’re a good structure, Charlie Brown: The distribution of narrative categories in comic strips. Cognitive Science, 38(7), 1317-1359. doi: 10.1111/cogs.12116

Monday, September 08, 2014

New paper: Building a better "comic theory"

I'm happy to say that I have a new paper (pdf), "Building a better "comic theory," in the latest issue of the journal Studies in Comics. In this one I critique the existing theories about "how comics are understood" and provide a framework for better research to be undertaken. Longtime readers of this blog will certainly recognize some of my advice for researchers, now presented here in a coherent fashion.

Here's the abstract in full:

Research on the understanding of ‘how comics work’ has grown tremendously over the past twenty years, with more articles and books emerging each year. Much of this research has discussed comparisons between comics and language, and/or has speculated on comics’ cognition. However, much of this research faces limitations, which hamper the seriousness of the endeavour and reflect the youth of this emerging field. This article points out these deficiencies that pervade theories about comics. These include inadequate background research, overly general and unsupportable claims, a lack of adequate evidence, and limitations for research methodologies. To address these concerns, I draw from over 50 years of research from linguistics and cognitive science to inform how the field of ‘comic theory’ can move forward. In particular, I outline two primary ways of progressing with this line of research: (1) explicit manipulation of the component parts of the structure used in comics and (2) cataloguing actual comics for various theoretically relevant phenomena. This data-driven approach is offered as a guiding vision for future works on the understanding of ‘how comics work’.



Full Reference:

Cohn, Neil. 2014. Building a better “comic theory”: Shortcomings of theoretical research on comics how to overcome them. Studies in Comics. 5(1), 57-75

Wednesday, September 03, 2014

New book on Australian Sand Narratives

Anyone who has read my new book or followed my work online for the past several years knows that I frequently talk about sand narratives created by Australian Aboriginals as a counterpoint to the structures found in the visual language of comics. I find these drawings made in sand to be extremely interesting and important for the study of sequential images and drawing in general.

In fact, learning about them has been pivotal in forming my broader ideas of visual language, which is why I included a whole chapter about them in my book.
I first learned about Australian sand narratives from my post-undergrad advisor Dan Slobin, who introduced me to David Wilkins, a linguist who had written about the sand narratives of the Arrernte. This led me to read lots of papers by anthropologist Nancy Munn about the similar systems created by the Warlpiri. I'm happy to say that seminal papers by both of these authors will appear in my upcoming edited collection on visual narratives, out in late 2015/early 2016.

However, for the past several years I've corresponded with Jennifer Green, who has actively been doing extensive research on sand narratives and how they interact with spoken and signed languages. Her work is amazing and thorough, and is wonderfully now available in a new book:


In the chapter on my book, I attempted to distill the information about sand narratives regarding the features of the graphics alone. However, as with images in comics, they rarely appear isolated, and are usually embedded within a broader system of speaking, drawing, and signing. It is these broader multimodal interactions which Jenny is most concerned.

If you are interested in how these systems work beyond the broad overview I present in my book, I highly recommend checking out Jenny's new book. It is extensive, enlightening, and is now the first comprehensive work discussing these systems. Hopefully it won't be her last.

Here's the synopsis:

"Sand stories from Central Australia are a traditional form of Aboriginal women's verbal art that incorporates speech, song, sign, gesture and drawing. Small leaves and other objects may be used to represent story characters. This detailed study of Arandic sand stories takes a multimodal approach to the analysis of the stories and shows how the expressive elements used in the stories are orchestrated together. This richly illustrated volume is essential reading for anyone interested in language and communication. It adds to the growing recognition that language encompasses much more than speech alone, and shows how important it is to consider the different semiotic resources a culture brings to its communicative tasks as an integrated whole rather than in isolation."

Thursday, August 28, 2014

An update: Talks of future past

I've got roughly a month left of my summer (yay quarter system!), and it's been a wild ride so far. I've given talks in Germany, Québec City, and Seattle, and by the time summer is over, I'll have given roughly 10 talks over three months!

To cap off this tour, I'm about to head back to Germany for a few more talks at the end of September. The first will be for the "Empirical approaches to comics" workshop in Berlin. They've recently posted their schedule online and it looks like a very fun line up!

After that I'll be speaking at some researchers' labs in other parts of Germany, before heading back to Berlin for the academic ComFor conference on comics. There I'll be giving the keynote discussing the relations between comics and the brain. (For those of you going, if you have requests for which research you want to hear about, let me know!)

Speaking of talks, I'm excited to say that my recent talk up in Seattle at Microsoft's FUSE Labs is now online! You can watch it with additional slides on their site here, but you can also just watch it below.

Finally, Happy Birthday to the "King" of comics Jack Kirby! He'd be 97 today.





Friday, August 22, 2014

"Bad anatomy" and body objectification in comics

Greg Land cover and redrawn version from www.themarysue.com
The recent uproar over the covers of Spider-Woman books by Marvel has lead to cries that they are both sexist and poorly drawn. I think that these covers, and especially the responses to them, make for a good opportunity to examine the idea of "bad anatomy" in drawings and how bodies are drawn.

Concern #1: Bodies are depicted "unrealistically" in comics

The first critique of drawings like this is that they show "bad anatomy." This critique is common of course to many comic artists—whether there are claims that their work is sexist or not. For example, Rob Liefeld is basically mocked on regular occasion for his drawing of bodies and feet to the point its become a culturally accepted meme that he's a bad artist (he's not).

From a cognitive perspective, none of these works are "bad drawings." As I argue in my recent book, The Visual Language of Comics, and in several articles, drawing is not now—nor has it ever been—about drawing "realistically." The idea that drawings should somehow mimic our perception and align with "the way things are in the world" is pure fallacy (I previously called this idea "Iconic Bias").

This can only be addressed by considering "what are drawings?" in the first place. Drawings are a collection of graphic patterns stored in the head of an "artist" who combines them to form depictions. People draw faces by combining their learned head patterns with eye, nose, and mouth patterns. Feet (including those by Liefeld) are drawn using feet patterns, as are everything that a person draws. The same is true of "cutting off" legs at the knees to imply they are behind a person—it's just a pattern stored in the head of the artist. I have several examples of these—from many well respected comic artists—throughout my book.

This has nothing to do with how things look "in real life." It is about the patterns that an artist has learned over time and the ways in which they then use them in drawings. Why do so many people feel they "can't draw"? It's because they haven't built up a "visual vocabulary" of these patterns to be proficient at drawing.

In this cognitive point of view, there is no "bad anatomy" in these works. These artists—including Liefeld—are merely using the patterns of their "visual language." And they are doing so with high "fluency." The same is true of Peanuts characters or The Simpsons, which are more cartoony representations—where's the outrage that they only have three fingers? Or giant eyes? Or have elbows that appear and disappear? The cognitive fact is that drawing is not about "accuracy" to the "way things look in the world," be it in comics, manga, airplane safety guides, or art from any culture of the world and any time in history.

So... what's the issue with these bodies then? It's an issue of societal opinions about the representations of bodies—particularly women's bodies. This is not an issue of proficiency or anatomy—they are not "bad drawings" and in fact these artists (including Liefeld) show extremely high proficiency.

Rather, it's the fact that the way that artists of the "American Visual Language" used in superhero comics draw women is offensive to some people. This is directly analogous to the way that various words in spoken languages are offensive to some people. The "visual vocabulary items" used to draw women are on par with calling women any number of distasteful words.

People must accept that these patterns are merely a part of the learned visual vocabulary within a broader "visual language" in which that artist partakes. And, of course, that visual language is embedded in a broader culture in which women most certainly are marginalized and objectified. In order for that artist to draw women differently, they would have to learn new patterns, in the same way that they would have to use different words to speak about women in less offensive ways (which I discuss in length in this post).

Note, we've seen this before. African-Americans in the early 20th century were drawn in wholly offensive ways as "tar babies." These artists were not necessarily trying to be racist, but they used a graphic pattern that was inherently racist—and they were embedded in a culture that was also inherently racist. As people realized this about the images, the culture reacted to change the visual language such that people thankfully no longer draw this way. This is directly parallel to ostracizing the various racist words for African-Americans in our spoken language.

Again, this is not an issue with "anatomy" or proficiency in drawing—it is about the sociocultural use of different patterned representations which may or may not be offensive to people.


Concern #2: The depiction of bodies in comics

This leads us to the second issue, which is that women's bodies are drawn to be objectified in comics. As it happens, we have some data about this...

Last quarter at UCSD, I taught a class on the "Cognition of Comics" where all my students did original research projects. One student, Bianca de la Garza, decided to study the sexualized depictions of both female and male bodies in 15 American superhero comics randomly selected from the 1940s through the Present. She coded bodies using a scheme that ranked the "sexiness" of whole characters and their body parts (chest, butt, face, arms, legs, hip to waist ratio, etc.) and assessed the feasibility of their poses. Her 1-7 scale ranged from a 1 being "unhealthy" both obese or undernourished (i.e., the Blob or Plastic Man), a 4 being "normal", and a 7 being grossly hypersexualized (i.e., gigantic muscles, enormous breasts, etc. to the point of disfigurement). Thus, a 5 or 6 would be a highly sexy but not disfigured depiction.

Now, going into this, we both predicted that, as is the cultural belief, women would be far more sexualized than men in their depictions. In fact, we were totally wrong. In almost every category, females and males appeared to have the same ratings of sexualization, and this relationship between genders did not change over time (though the sexualization of certain body parts did change over time).

(Note that these findings differ slightly from those in other studies, discussed below.)

Interestingly though, across all time periods, the numbers of women represented in our sample were vastly lower than men: Women were underrepresented in the books (including some where a woman was the title character!). While we did not code for the content of the stories, my suspicion is that the underrepresentation of female bodies (i.e., female characters) also correlates with women playing less significant roles in those books, which of course reflects a marginalization of them.

So, our data showed that while women's bodies may indeed be drawn in objectified and sexualized ways, the same is true of men's bodies. Furthermore, I'll go so far as to say that this is not an issue about comics, but about the depiction of bodies—regardless of sex—in media throughout our culture. How often do you see ugly people of either sex on television or movies? Did I miss the uproar about the sexism about Chris Pratt having to bulk up his body for Guardian's of the Galaxy? His shower scene was clearly a gratuitous look at his body, and Dave Batista spent that entire movie without a shirt on! (Of course, as in our sample, there was also only one woman on that team, and she wasn't the lead star).

Now, I happen to be particularly sensitive to these issues because my parents have written and published books about eating disorders and body image issues for over 30 years. They were the first to write a book for laypeople about bulimia, my mother was the first openly recovered bulimic on television, and my father wrote the first book about men's body issues (which happens to be dedicated to me).

While the objectification of women's bodies in our culture is certain true, it is also true that men's bodies have become increasingly objectified and scrutinized over the past decades. The question for comics then, is whether it's acceptable to maintain a visual language where all bodies are shown in this manner, or whether people want to push for "visual language reform" along the lines of changing American culture's visual vocabulary.



-------------------------

Addendum: Two other studies have looked at bodies in American superhero comics (citations below), and here is what my student Bianca noticed about them: Cocca finds that many women are depicted in unfeasible poses. However, her analysis largely ignores mens bodies. Healey and Johnson meanwhile found that male bodies were closer to "average" than females', which were more underweight, according to the "stats" for male and female bodies for Marvel characters by calculating their BMI. However, this study didn't look at actual drawings (just character stats), nor did it differentiate between body types—an underweight character could be malnourished and an overweight figure could be fairly muscular (a problem with the BMI in general).

If you want to explore this topic more, I encourage you to do your own corpus study!

Cocca, C. (2014). The‘Broke Back Test’: a quantitative and qualitative analysis of portrayals ofwomen in mainstream superhero comics, 1993–2013. Journal of Graphic Novels and Comics, 1-18.

Healey, Karen, and Terry Johnson. Comparative Sex-Specific Body Mass Index in the Marvel Universe and the "Real"World.

Tuesday, August 19, 2014

Violating your drawings!

Here's a fun blog post by Chris Schweizer on spotting "compositional tangents" while drawing comics. For those unfamiliar, Chris does the excellent "Crogan's Adventures" series (Crogan's Loyalty, Crogan's March, and Crogan's Vengence) which I highly recommend.

So, what are these "tangents" that Chris is talking about? He describes tangents as "when two or more lines interact in a way that insinuates a relationship between them that the artist did not intend."

These include things like when one line runs into another line (as in the example image, taken from Chris's article), or when lines all come to a single point unintentionally. He lists quite a range of them.

While it isn't true of all of them, what he's describing are essentially "illegal" line junctions of "graphic structure"—the component part of the drawing system that guides how we produce lines and shapes. The tangents that Chris discusses aren't horribly bad junctions, which I'll talk about below, but they are illegal combinations nonetheless.

In my book, I've compared the way that lines combine in drawings with each other to the way that sounds combine together in speech. For example, in English, it's illegal to combine a "t" and "g" sound at the start of a word or syllable.

Similarly, the way that lines combine in line junctions is how graphic structure creates shapes, and different junctions imply different meanings. For example, a T-Junction looks like a "T", and implies that one object is placed in front of another object. You can see some of these in the lines of the barn above, and of the interaction between the barn and the background shrubbery. The "T" shaped junction between these lines gives the meaning that the barn is in front of the background.

Certain combinations between lines thus lead to "illegal" junctions analogously to illegal sound combinations in speech. For example, the "continuous line" junction shown in the image above is weird because one line runs directly into another as a junction. You can very much mess up an image by altering its line junctions, as in the image below, which comes from my book.


I always like pointing these out, because they nicely fly in the face of the idea that you can't make an "ungrammatical" drawing. Something is clearly wrong here! These junctions are discussed at length in the excellent book, Art and Representation by John Willats.

So, most of these problematic cases reflected in Chris's article are problems with line junctions, and, as he says, they are easily remedied. The easiest solution is simply to make one thing overlap another—to create a T-Junctions—so that one thing appears to be in front of another. Go read the article!

Wednesday, August 13, 2014

Visual style and visual morphology

While I was on the University of Bremen campus in Germany last month, I stumbled on a very appropriate poster hanging in the linguistics department (right) which sponsors a nice thought experiment about the relationship between conventionalized graphic "morphology" (like motion lines, thought bubbles, or lightbulbs/gears/hearts floating above people's heads) and the visual "style" they are represented in.

Of course, what caught my eye was that it plays on the "lightbulb above the head" convention, which I call "upfixes." In my book, I've argued that upfixes form a class of "visual morphology" where some sort of object floats above a person's head to create an even larger meaning. These can be gears for thought, hearts for love, ?! for surprise, lightbulbs for inspiration, and many others.

What caught my eye here is not just that it playfully evokes an upfix by having a person holding a lightbulb, but that the act of holding it doesn't interfere with the meaning. In fact, the holding of the lightbulb somewhat allows for the fact that it's a photo to be ok.

For example, we might suspect that having a realistic photo of a person trying to evoke a drawn convention might look weird. And, of course, someone holding a lightbulb does seem a little weird, but not horribly bad. We still get the sense of the "inspiration" meaning out of it.

However, compare this to the poster when we fully get rid of the hand (left)—now resulting in a realistic lightbulb floating above his head. To me at least, this looks a little weirder, even though it's actually more "accurate" to the upfix!

My suspicion is that we're more ok with the use of visual morphology in realistic photos when they retain their "realism" using the physics of the "real" world. However, doing the same thing when the morpheme is more similar to its actual graphic equivalent is weirder (i.e., just floating), because it violates the principles of the realism? (I dunno... Maybe it would be better if the lightbulb was lit up?)

I don't think that it worked all too bad with the upfix cosplay that I stumbled on at WonderCon earlier this year with the exclamation mark, but I'm not quite sure how far we can push this... Clearly, drawn hearts replacing eyes ("eye-umlauts") are a convention to mean love. I think it'd just look strange to have hearts placed over someone's eyes in a photo, right? What about holding actual gears above someone's head?

There seems to be something interesting here about how far one can go with non-iconic elements of visual morphology and their "stylistic" realism. Perhaps this is also why superhero comics don't use morphology like eye-umlauts or upfixes: they often attempt to maintain some degree of "realism" despite their fantastical powers and those associated conventions (like motion lines, x-ray vision, etc.). This seems like a corpus study and/or an experiment just waiting to happen...


Wednesday, August 06, 2014

Page layouts with the Golden Ratio

I was recently sent a link to this blog post from back in May that discusses the "Golden Ratio" in the creation of comic pages by Bizhan Khodabandeh. He has some interesting ideas about how basic geometric shapes can underlie page composition in a way that's pleasing to the eye. He therefore advocates using the "rule of three" to divide up pages and space, coming at the issue from a very "graphic design" perspective.



I'm not sure if I'm convinced that these pages are aesthetically more appealing or easier to understand than pages that may not follow these exact ratios, but the idea is interesting nonetheless. Following such ratios may lead to some unintended consequences though... by so rigidly following the underlying shapes in the example above, it looks like the bird's legs are at an awkward angle such that the body weight wouldn't be supported by them. But, this is a very minor issue I'd think, and the ideas are definitely worth checking out.

Essentially, what the author is talking about is the relationship between External Compositional Structure (ECS)—the physical relations of panels on a page—and Internal Compositional Structure (ICS)—the organization of elements within a panel. The ideas of making these elements interact has long been a focus of both comic creators and comic theorists (Groensteen talks about this a lot).

What I find interesting here specifically is how far it pushes this interaction in treating the page as an aesthetic canvas. In research on ECS specifically, one of my students coded American superhero comics from the 1940s through the Present and argued that page layouts have been growing in their treatment of "pages as aesthetic canvases" rather than treating them like containers of "panels in a flow of information."

That is, authors are becoming more sensitive to how the page as a whole works as an aesthetic whole. The ideas in this blog post make such ideas very explicit.

Monday, July 28, 2014

Recap: Cognitive Science 2014

This past weekend I got to enjoy the annual Cognitive Science Society conference in Québec City, Canada. Unfortunately, this year's conference conflicted with ComicCon, but in the end it was definitely worth heading up to Canada than just going a few miles down the road here in San Diego.

I had several presentations throughout the weekend, but the highlight was the "Cognition of Comics" symposium on the first morning. Over the past few years, I've started collaborating on visual language research with a few other researchers, and I assembled everyone together to talk about our work.

I offered both an opening overview of my/our line of research, and at the end of the symposium I presented a wides scale view of how cognitive science issues can be studied using comics. In-between, my collaborators presented. David Wagner (University of Stavanger) and Tom Foulsham (University of Essex) both presented the work they've been doing using eye-tracking to see where people look while reading comics. Then, John Drury (Stony Brook University) presented work of ours looking at brainwaves in the comprehension of visual narratives, math, sentences, and music. This work seeks to explore whether the same brain mechanisms are involved in processing all of these domains.


Beyond the incredibly interesting talks themselves, as far as I know, this symposium is the first time multiple empirical psychology studies about the cognition of comics has ever been put together. To me, it felt like the starting point of the nascent field of visual language research—very exciting!

This feeling was only reinforced because we had ongoing discussions about research and comprehension of visual language for basically four straight days. Looks like there's going to be some great research up ahead from all of us, and a lot of work to be done! Because of this, I imagine we might look back on this conference as a seminal point in the history of this research.

The other special thing about this conference was that my mentor, Ray Jackendoff, was awarded this year's Rumelhart Prize, which is the "Nobel" of the cognitive science field. It's hard to think of a scientist who is more deserving. The award not only honors his extensive scholarly contributions to areas across cognitive science (language, music, perception, consciousness, social cognition, etc, etc.), but also his teaching and mentoring. This is also quite deserved, as I can say as one of the few individuals to graduate with a PhD with him as my primary advisor.

As part of the award, Ray gave an impassioned talk about the need and benefit of "theory" in the cognitive sciences. I recorded video for the talk, and we're going to try to get it posted online for all to see. Link to come.

Sunday, July 20, 2014

Rules of Emoji

A friend passed along this article recently which describes research by Tyler Schnoebelen that explores the rules people use when writing with emoji/emoticons. When people hear that I work on "visual language" they often think of things like this. However, Schnoebelen actually did study these things, scouring through roughly 500,000 tweets and analyzed the types of productions that people made.

For example, he noted that people tend to place emoji at the end of clauses and sentences rather than the beginning. This makes sense, because people won't want to divide up their syntactic clauses. Instead, they'd prefer to use emoji like punctuation to divide up those clauses.

There also seems to be at least some linear causality when people are trying to communicate in emoji alone. The article notes that a story told through emoji would have to go in sequence, like this:
But, rearranging the order creates a different story, possibly less comprehensible:
This type of ordering could possibly use the rules in my visual narrative grammar, but in fairly stripped down and choppy form. Scrambling the order of units though is a common technique we've used before in studying the structure of sequential images.

These are particularly interesting for me because this type of things is what people often mistake my research for. However, emoji are not actual "visual language" as is laid out by my theory. Rather, they are at the intersection of visual language with writing—a conversion of sound into graphics (a learned synesthesia). They can act like "visual gestures," supplementing or enhancing the expressions of the text. Or, they can be like a pidgin, a hybrid communicative resulting from the intersection of rules of different systems, often using combinations based on basic rules of meaning without a complex grammar.

Nevertheless, emoji are particularly interesting because they still can reveal various preferences that people have for ordering meaningful information that is likely shared across various domains. For example,  another thing discussed in the article is that people will generally show an emotional state first, then show the cause of it, as in the laughing-monkey-poo example above and this one, where sadness precedes the reason why (a broken heart):
These examples use a structure common across communicative systems, which is to express the Agent (the doer of an action) prior to the Action. In this case, the Agent is sad, and the sadness is caused by the action of a broken heart. Often, this pattern will have a Patient (a receiver of an action) as well, in a canonical Agent-Patient-Act order. This pattern occurs very frequently in both full languages (such as Subject-Object-Verb ordered grammars), and in communicative systems that lack a full grammar (such as when non-sign language speakers are asked to communicate with only gestures).

I think there could likely be some interesting work done using experiments that ask people to communicate with emoji alone. That might be a nice follow up to Schnoebelen's work. He's done the corpus analysis of looking at what people have done in naturalistic expression. An interesting next stage would be to test various rules in experiments.

So... are emoji "visual language"? Not really in any full blown sense. However, they can tell us about verbal and visual languages, and the rules of communicative systems more generally.


(Wayback machine note: A long time back I did this related post about emoticons and the brain)