Wednesday, December 07, 2011

Eye movement in reading comics

I've posted a few studies that have looked at how people's eyes move across comic pages (here and here), and I recently found another. This short study looked at when people's eye movements skip panels or go back and re-read them. 

They found that people spend more time reading panels with text than with just images, and that panels without text are more likely to be skipped and to be read with peripheral vision. Unusual panel arrangements (i.e. non-horizontal then vertical arrangements) also possibly led to jumping over panels (as was found in another study as well). After skipping these panels, participants then backtrack and re-read them.

These findings are consistent with previous studies that compared the eye-movements of expert and non-expert comic readers. Non-experts tend to focus more on text and read more erratically throughout a page. Experts tend to read more smoothly and focus more on the images. 

General studies like this are interesting, though I'd really like to see more studies that specifically target specific issues. Are there particular features of page layouts that motivate skipping panels? Are there features of layouts that impede on the actual comprehension of panels? Once we get beyond these very basic sorts of "what do eyes do generally" studies, we can really start exploring how looking at eye-movements can tell us about the comprehension of comic pages.

Chiba, Shinichi , Takamasa Tanaka, Kenji Shoji, and Fubito Toyama. 2007. "Eye Movement in Reading Comics." In Proceedings of the 14th Annual International Display Workshops, 1255-58 

Wednesday, November 23, 2011

Attention and comic panels

Craig Fischer has a nice article over at The Comics Journal about how panels focus attention, particularly focusing on the work of Jack Kirby. He nicely keys in on several techniques that authors (like Kirby) use to highlight certain aspects of a panel over others. For example, putting things in the foreground vs. background, thick lines vs. thin lines, or focusing on people vs. objects.

He then goes on to create an interesting taxonomy of ways that content connects with a narrative, and whether the focal and background elements are done in a common style. As a descriptive taxonomy, I think it works pretty well.

At the end of the piece, Fischer wishes there was more empirical work on how people read comics, especially with eye-trackers. Apparently he hasn't been reading this blog much! Amongst the many studies I've reviewed here about comprehending sequential images, there have been some eye-tracking research on comics that I review here and here.

Also, while my empirical work has mostly focused on how sequential images are comprehended, my theoretical work has looked at the capacity of panels to convey attention for many years. For example, I discuss it in this blog post, as well as in my article, A Visual Lexicon (pdf).

My approach to attention has focused less on the individual aspects of a panel's features, and more on how the panel as a whole acts as a "window" onto a scene. The panel then simulates the same type of "window" on the fictitious world that attention does in our visual perception. As I said in that blog post:
"Most of the time though, panels serve to exclude all relevant information except for the elements that need to be focused on, or at least clearly distinguish what is relevant from irrelevant. This lets panels provide a graphic manifestation of this mental "spotlight," allowing the author to control that attention instead of the reader's wandering eyes (which is one of the reasons I formally call panels "Attention Units")."
However, there's much that could be learned by studying the combination of the types of attention that Fischer talks about (those visible in a panel) and those that I talk about (how what is visible connects with what is not visible, or to other parts of a narrative).

Monday, November 14, 2011

Take my online comics experiment!

At long last, I have another comic experiment ready to go that needs your help, dear reader! This survey will help us prepare our next study looking at how the brain comprehends comics, and your help would be greatly appreciated. It should take roughly 15 minutes and involves reading comics and giving a basic rating for how much they make sense. Participation enters you in a raffle for a $50 gift certificate to Best Buy.

UPDATE: This survey is now closed. Thank you for your participation! If you would like to participate in future experiments, please email me.

Wednesday, October 26, 2011

A checklist for decent scholarship

I've read a ridiculous amount of research on the comprehension of sequential images the last few years. Many people have written papers about this topic, often from many different disciplines. While I can respect that not everyone will aim for the linguistic and psychological approach that I use (and nor should they if they have different intents), there are several pet peeves that I found repeated over and over that make me feel like just disregarding what people write.

So, here's a checklist for getting me to take your scholarship on comics seriously:

Get your names right. I once read a paper that was cited as an "authoritative" source by another book only to find it said the author of Calvin and Hobbes was "Bob Watterson" in a trivial throwaway line. I could maybe forgive a spelling error or something accidental, but not knowing that his name is actually Bill made me lose respect for the entire paper and regard the scholar as someone who was a "tourist" in this research.

Get your basic facts right. An otherwise decent article commented that Chris Ware's Acme Novelty Library was "computer-generated." It's actually done by hand. Such a mistake could easily be remedied with a Google search. And, again, this was a trivial descriptor that was unnecessary for any part of the discussion.

Do your background research. Often I read papers where they only major work of scholarship about sequential image comprehension mentioned is McCloud's Understanding Comics. As important as it was for establishing this line of thought, it is not the only work out there. Or, if they want to talk about cognition, they'll cite a textbook written over 30 years ago. This just isn't acceptable. One of these days I'll write a huge paper reviewing all the theories and experiments that I've found that have ever looked at sequential images (and there are more than you'd think... I've been painfully remiss in updating my bibliography and I'll be the first to say that there's probably more out there that I haven't found yet). Until then, do your own research beyond a cursory job. The same goes for any topic of research or else you'll get a response like this.

Say something novel. It's amazing the amount of papers that I read that merely regurgitate McCloud's ideas of panel transitions and closure without adding anything new. At best, they often just reinterpret his same ideas and draw a connection between them and something in another line of thinking. But, they don't add anything to what he said. Almost ten years ago, I just about doubled the amount of transitions McCloud had before abandoning panel transitions altogether.  So, even if you're working with transitions, I know there's more to say than he did (and probably what I did too). Truly, if you can't say something new, why bother saying anything? (AND... why should anyone read or cite your paper??)

Seriously though... shouldn't all these things be part of basic research and paper writing? Why then are they so rampantly disregarded when talking about the structure of comics? For a long time, scholars of comics felt the need to justify such research. However, the best way to convince people to take this work seriously is to actually do serious scholarship. 

Tuesday, October 18, 2011

On tap...

Thank you to everyone who came out for my talk last week at the New York Comic Con! I had a great time giving the presentation, so I hope it was as enjoyable for all you in the audience.

I have a few blog posts planned for the coming weeks if I can find the time to finish them off. As usual, about a dozen projects are being worked on right now. Included among those is one that we will soon be launching an online experiment for. So, if you'd like to help with some comic research, watch this space over the next few weeks!

Thursday, October 06, 2011

New York ComicCon 2011

Next Saturday on October 15th I'll be speaking at the New York ComicCon for the first time. Here's my blurb from the program:
4:00-5:30 Comics Studies Conference 6: Understanding Comics and the Self—Neil Cohn (Tufts University) discusses several psychology experiments measuring reaction time and brainwaves that contribute to our understanding of what goes on in the brain when a person reads a comic and reveals that the understanding of comics involves a complex negotiation between a hierarchic system of narrative and the construction of meaning. [...]
I'm quite excited about this talk because it looks like I'll be presenting brand new data from my latest study of comics and the brain. Hope to see you there!

Thursday, September 29, 2011

Development of drawing abilities

The book Making Sense of Children's Drawings by John Willats puts forth a compelling theory of how kids learn to draw, and the course of that development.

To Willats, drawings link up to our mental conceptions of how things look in the world, thereby rejecting a view that says drawings are entirely based on what we see. A great example of this is when children are given dice and told to draw them. Instead of drawing them as they see them, they draw dice with all six sides, which would be impossible to see.

Willats also provides great detail on the origin of the "don't copy" trend of instruction in drawing (which he, like me, is highly critical of). As he describes, this came originally from the 1800s educator Franz Cižek, based on Romantic ideas that children had a pure "inner creativity" that needed to develop unspoiled by imitation from external influences.

As he nicely points out, this doctrine is largely not reflected by what children actually do. Indeed, closer inspection of Cižek's own students show a consistent group style. They were copying between each other, just not from him.

Also, his general trajectory for learning to draw runs like this:
  • 1-3 yrs: dots, lines, regions. Scribbles denote whole regions of space, not necessarily just random uncontrolled lines.
  • 2/3-8 yrs: Bounded areas depict regions and volumes. Round, long regions denote round long volumes, while long or round regions show flat volumes.
  • ~6-10 yrs: Regions are used as as picture primitives to denote faces rather than volumes. However, lines still denote boundaries of regions, not the contours of shapes.
  • ~6-8 yrs: Regions as volumes, but compensated by more modifiers, resulting in "having a smooth outline" (threading); denote regions n the visual field (starts approaching lines as contours)
  • ~8-10 yrs: Finally, lines are used as picture primitives (instead of using lines for regions). Lines are finally used as contours, as evident by line junctions used for occlusion and foreshortening.

The one drawback to this approach is that, despite his critique of the overall trend against copying, his developmental trajectory does not incorporate the effects of imitation on drawing. This may not be possible for him though: there simply doesn't seem to be enough data, looked at through the right perspective, to offer his model much more (true both of when the book came out, and now).

However, overall, I highly recommend this book to anyone interested in how children learn to draw.

Willats, John. 2005. Making Sense of Children's Drawings. Mahwah, NJ: Lawrence Erlbaum.

Tuesday, September 20, 2011

Graphical Abstracts

A friend of mine passed along this interesting link today. The academic publisher Elsevier looks like it's now accepting "graphical abstracts" for scientific papers in journals:

A Graphical Abstract should be a one-image file and should visualize one process or make one point clear. For ease of browsing, the Graphical Abstract should have a clear start and end, preferably "reading" from top to bottom or left to right. Try to reduce distracting and cluttering elements as much as possible.

Also interesting is that they expressly specify how they want the image to be "read." Now I'm curious what they'd think to an abstract using sequential images...

Wednesday, September 14, 2011

What is the human language faculty?

My mentor, Ray Jackendoff, has a new article out in the journal Language that mentions my research (as well as has some illustrations by me):

Jackendoff, Ray. 2011. What is the human language faculty?: Two views. Language 87(3):586-624

The piece explores the biological foundation of our capacity for language, and what components of cognition contribute to language understanding. My work comes in because he points out that several cognitive capacities involve the hierarchic organization of structures, including language, music, vision, events, and, yes, the visual narrative in comics. Here's the abstract:

In addition to providing an account of the empirical facts of language, a theory that aspires to account for language as a biologically based human faculty should seek a graceful integration of linguistic phenomena with what is known about other human cognitive capacities and about the character of brain computation. The present discussion note compares the theoretical stance of biolinguistics (Chomsky 2005, Di Sciullo & Boeckx 2011) with a constraint-based PARALLEL ARCHITECTURE approach to the language faculty (Jackendoff 2002, Culicover & Jackendoff 2005). The issues considered include the necessity of redundancy in the lexicon and the rule system, the ubiquity of recursion in cognition, derivational vs. constraint-based formalisms, the relation between lexical items and grammatical rules, the roles of phonology and semantics in the grammar, the combinatorial character of thought in humans and nonhumans, the interfaces between language, thought, and vision, and the possible course of evolution of the language faculty. In each of these areas, the parallel architecture offers a superior account both of the linguistic facts and of the relation of language to the rest of the mind/brain.

narrow faculty of language, recursion, parallel architecture, Merge, Unification, lexicon, consciousness, evolution

Thursday, September 08, 2011

Segmentations in visual narrative

Gernsbacher's 1985 paper "Surface information loss in comprehension" is an important article on the comprehension of sequential images, and one that has informed much of my current research. It is based on her dissertation, and describes several experiments.

Overall, Gernsbacher had participants read the Mercer Mayer book Frog, where are you? to question whether people can accurately recall the exact surface images in the story, or if (like language) they are only able to retain the gist of meaning.

First, she asked participants to read this "picture story" and choose where they would divide it into parts. They simply drew lines between images where they felt that one episode ended and another began. Overall, she found that people greatly agreed on where these boundaries between segments were placed.

She then asked another group of people to read the stories, but the composition of certain images were flipped horizontally. These images either came before or after the boundaries that people agreed upon in the previous experiment. She found that people had a harder time accurately remembering the horizontal composition if the image came after the boundary as opposed to before it. This provided evidence that people were building up context throughout a segment, and that the start of a new segment incurred a cost on memory.

These experiments were important for several reasons. First, it confirmed her hypothesis that people mostly retain the gist of meaning and not the surface information of images. Given that people's comprehension did not appear overly damaged by flipping the composition of images, it could be pertinent to discussions of how much impact is really made by the left-right composition of images, such as in the 180º rule.

However, more importantly, these experiments showed very strong evidence that people group images together into segments. This poses a problem to theories like McCloud's panel transitions, which  envision no stopping point for linear transitions: they keep going on and on throughout a visual narrative (either linearly or promiscuously between multiple panel relationships).

Rather, this experiment shows that people have some intuitions for dividing up visual narratives into segments (what I called in my book "visual sentences"), and that moving between those segments incurs a cost to comprehension.
Gernsbacher, Morton Ann (1985). Surface information loss in comprehension. Cognitive Psychology, 17 (3), 324-363 DOI: 10.1016/0010-0285(85)90012-X

Saturday, September 03, 2011

Waning days of studenthood

Tuesday will begin what looks to be my last year of grad school, which means my last year of being a student. Yikes! That means I'm currently balancing finishing my projects, helping teach classes, writing/revising papers, and looking for what's next. Should be a wild semester!

I have quite a few projects underway right now, starting with my second brainwave study looking at the comprehension of sequential images. I got some good data on this experiment over the summer with a "reaction time" experiment, so now its time to stick electrode caps on people! If all looks good with troubleshooting, we could be up and running this week.

Also, coming up in October I'll be making an appearance down at the New York ComicCon, giving a talk on my research on Saturday, October 15th. As the date gets closer, I'll offer more info.

Sunday, August 14, 2011

What do you do with a comic?

Based on some disagreements I've had with people lately, I'm curious what the general populace thinks. Please take my poll (and pass it on!):

Also: If you choose "other" for either one, please feel free to say what you think in the comments. Thanks!

Thursday, August 04, 2011

Brainwaves for non-sequitur visual sequences

Here is another repost of a review I did awhile ago (1/22/09).

This study examines the neurocognitive processes involved with comprehending a series of pictures, like in comics. The experimenters pulled frames from an animated movie to create static picture sequences. There were two possible endings for each sequence: one with a normal ending, and one with a non-sequitur panel that did not make sense.

Comparison of these sequences used a technique called "event-related potentials" (ERP) that examines people's brainwaves with an EEG recording. The electrical field is measured off the top of the scalp through an electrode cap (like in hospitals), and by averaging out the noise at the critical point (the "event" — here the last panel) it can give you a nice smooth waveform that can tell you about the nature of the cognitive process. Unlike fMRI, ERPs don't tell you much about "where" in the brain things happen, but they do tell you a lot about "when" and a little about the nature of the process.

In this case, your brain distinguishes the difference in processing at less than half a second. The result was a "negative" deflection of the waveform roughly 400 milliseconds after the final panel appeared on the screen (panels appeared one-by-one). These waveforms are from the frontal right part of the head:

The BLUE line represents the normal sequence ending, the RED line the non-sequitur ending. Note that the lines separate and there is a bump labeled "N400" that shows the processing difference (negative is up here). Because of the separation, we can tell that the brain is working harder to process the non-sequitur panel. If it was treated the same, the lines would stay together, like at the beginning of the waveforms.

This N400 also appears in language under similar conditions: where the brain is working harder to integrate semantic information into a meaning, though with language it appears in different locations on the scalp (more back of the head than front). In fact, the first paper that found an N400 for language used this same manipulation: comparing normal and incongruous words at the end of a sentence.

Unfortunately, more experiments of this sort have not really been done with sequential images. Fortunately, it's only a matter of months until I do more. Phil Holcomb, one of the authors, is also one of my advisors. My upcoming projects will be doing these types of brainwave studies using more targeted manipulations of the visual grammar.

EDIT (8/4/11): I have now done a study examining visual narrative structure and am soon going to do several more!

ResearchBlogging.orgWest, W., & Holcomb, P. (2002). Event-related potentials during discourse-level semantic integration of complex pictures Cognitive Brain Research, 13 (3), 363-375 DOI: 10.1016/S0926-6410(01)00129-X

Wednesday, July 27, 2011

I own a metaphor!

I am now the proud owner of an original piece of Drew Weing art! Drew is selling pieces from his excellent works,and I chose this one:

I like this one so much both because it reflects a feeling common to both comic authors and academics (of which I'm both), as well as it shows a great conceptual metaphor. Here, the head is mapped to the idea of a container that bursts when faced with emotional overload, just like a pot of water bubbles over when the water boils. In fact, I discussed this metaphor for this very comic last year in this post.

So, if you're interested in helping out a great author and buying some great original comic art for ridiculous prices, head on over to his site!

Sunday, July 17, 2011

Taking a year off

Astute followers of this website will have noticed a conspicuous lack of an announcement of my ComicCon talk this year. In fact, I am taking a break from the Con this year and instead presenting at a cognitive science conference this weekend in my current town of Boston instead of going to my hometown of San Diego. This is actually the first time in ten years I won't be at ComicCon, the last time being when I was living in Japan in 2001. So, I hope everyone has a great Con and perhaps I'll see you there next year!

Otherwise, research is still plugging along with vigor. I have several papers being prepped for publication, as well as several studies progressing along nicely. Hopefully within the next month or so we'll have another online study for people to take too.

Monday, July 04, 2011

Looking at Comics in the Brain with Lasers!

Nagai, Masayoshi, Nobutaka Endo, and Kumada Takatsune. 2007. "Measuring Brain Activities Related to Understanding Using near-Infrared Spectroscopy (Nirs)." In Human Interface and the Management of Information: Methods, Techniques and Tools in Information Design, 884-93. Heidelberg: Springer Berlin

I had originally reviewed this article several years ago back in 2008, but after recent research of my own I think it's worth revisiting...

This study used near-infrared spectroscopy to measure blood flow in the brain while people read comics. This technique emits infrared light into the scalp to measure where blood flows in the brain, which can thus indicate the brain regions involved in various behaviors.

They compared normal comic strips with fully scrambled strips of random panels. They then compared the normal strips with tasks asking people to either pay close attention to the strips and report what they found funny or to just read them passively. They found that there was greater activation in "the left prefrontal lobe region is activated when people actively try to understand the comic stories and to memorize their contents for reporting in the future."

As I reported last time, there are several problems with this study, such as the number of stimuli (only 6 strips) and their population (13 people). Comparatively, my last brain study used 160 stimuli per trial (720 strips total) and 24 participants.

However, the areas of activation that they did find is interesting. The prefrontal lobe in the left hemisphere ("Broca's Area) is associated with the processing of grammar in language, and the authors specifically point out that the areas the found for comprehending comics may overlap with this region.

Previous research has also found that this area plays a role in the comprehension of comics. Indeed, research of my own has been more fine-grained than these previous approaches, and has been finding hints that this area is active as well.

So, while this study may have great limitations, it may have been providing some early precedents for some important understandings about the comprehension of comics and the brain.

Tuesday, June 14, 2011

9 years... 1 to go?

As of May 31st, this website has now been up for 9 years! Woo! A couple notes...

Yesterday I formally proposed my dissertation, which means I am hopefully on track to finish my program by the end of next school year. That means I'll be running experiments from now until hopefully around October gathering data. So, keep an eye out for online studies coming soon!

Also, a very big congrats to one of my advisors, Phil Holcomb, who has recently received a prestigious MERIT award from NIH. This is a Method to Extend Research In Time (MERIT) "to provide productive investigators with a history of exceptional talent, imagination, and with a record of preeminent scientific achievements the opportunity to continue making fundamental contributions of lasting scientific value." It's a very big deal, since you can't be nominated for it — they just select you as one of NIH's most outstanding investigators. It's a huge honor and we're all very proud of him!

Monday, June 06, 2011

"180º Rule"... not so much

This is a review of an experiment that tests the "180º Rule" of film editing using eye movements, and finds that no evidence for negative cognitive effect is found.

The 180º Rule claims that in film editing, when showing two characters on the left and right of a shot, it would be confusing if the next shot reversed the perspective so that the characters end up on opposite sides. Filmmakers often dance around this by using over the shoulder shots that keep characters constant to their location in the frame.

The experimenters filmed two people having a conversation sitting around a table with a constant background from all angles. They cut the conversation into 22 shots and varied the number of correct vs. reversed-angle (180º violation) shots there were. This video was then shown to participants whose eye movements were tracked, measured from the "starting point" of where their eyes were located when the previous shot ended.

The results showed that eye movements were determined almost wholly by tracking who was speaking in the frame — the agent of the shot — no matter where they were located in the frame. The results showed no evidence for confusion at 180º Rule violations, nor did it show any evidence that participants were "mentally rotating" the scene to make up for those reversed angle shots.

In other words, all claims about the ill effects of 180º violations were not confirmed. They take these findings to indicate that editing rules do not cause confusion or ruin a scene's representation, and that the content of the expression overrides the way it is represented.

ResearchBlogging.orgGermeys, F., & d’Ydewalle, G. (2005). The psychology of film: perceiving beyond the cut Psychological Research, 71 (4), 458-466 DOI: 10.1007/s00426-005-0025-3

[Originally posted on 3/16/09]

Tuesday, May 24, 2011

Cross-cultural cognition via French Comic Translation Censorship?

I recently found this very interesting post about the "censorship" of French translations of Marvel Comics. Apparently this will be a chapter in a new book, but the examples they have posted (some relinked on the side here) are very interesting from the point of view of cross-cultural psychology and meaning.

While the post shows other types of censoring, the most overt things to me in the examples they show have to do with motion and action:

On the left is an originally American version of this page, while on the right is the French version of the same page. As you can easily tell, the biggest changes (besides the text) are the erasure of the impact star (the flashy star-shaped thing showing impact) and various speed lines (showing motion) and focal lines (emphasizing the impact point).

The post describes that this censoring took place to make these dynamic pages more "palatable" to children, as deemed by the government. However, I wonder if some other things are going on here as well.

First off, the lack of impact stars is interesting because it actually changes the meaning of the images. Daredevil and Captain America are no longer overtly strike each other. Without those stars, they may be interpreted as acrobatically dodging the blows. The impacts are no longer apparent.

Secondly, these changes may be a subtle way to try and make these American books appear more like native French comics. Several years ago, another presenter on a Comic-Con panel with me presented some interesting comparisons between French and American comics showing that French comics generally tried to show motion with few or minimal speed lines. The effect of erasing them in these pages makes the images closer to this native style. In other words, they are trying to translate the American Visual Language closer to French Visual Language.

As I was curious back at that presentation, I wonder if there can be a broader cross-cultural study done on this phenomenon. There have been several studies of "Paths" in linguistics and psychology, and speed lines are an overt manifestation of these types of paths. I am curious if:

1) American and European comics do in fact substantially differ in their proportion, usage, and type of motion lines.
2) If these differences reflect aspects of deeper issues in cognition related to the comprehension of path actions

Targeting these issues specifically could be an insightful way for using comics to study deeper aspects of cross-cultural cognition.

A broad look at a corpus of American and French comics could easily get to this issue. (My current corpus is lacking a good amount of French comics... if any of you French publishers are listening and feeling generous...). However, a follow up study looking at the Marvel translations and tabulating the proportion of these same items deleted could see if they truly are trying to translate the visual language in addition to the written language.

Sunday, May 15, 2011

Summer in spring!

As of Friday, I have officially entered Summer Break (here in a very rainy New England Spring), which means I'm hoping to have lots of interesting projects going soon. One of my goals is to resume blogging more frequently, at least a post a week. I think this should be doable given what I have lined up.

Most of my summer research is going to be devoted to running experiments in the laboratory here at Tufts, though I'll also be posting some online studies in the coming weeks. So, either watch this space, or keep an eye on your inbox if you're part of my "experimentation email list."

Monday, May 09, 2011

Comics as a Binary Language

The paper "The Comic as a Binary Language" by J. Laraudogoitia examines the structure of comics by converting the contents of panels into binary code. Coding a broad number of Eurpoean comics, a panel holding the protagonist of a story ("lead character") is given a "+" while a panel without is given a "-". The author then uses a series of computations to examine the regularity of sequences where the protagonist does or does not appear, or if there is constancy to the amount that they appear througout a book.

The results show that there is a quasi-regularity to sequences that feature the protagonist or don't feature the protagonist. That is, there are "runs" of sequences with protagonists, then runs without.

While interesting for coming up with a positive result — and very creative for applying computational methods to comics (somehting I don't think has otherwise been done), I find numerous problems with this paper.

First, why should we assume that Protagonist vs. Non-protagonist is a meaningful binary juxtaposition? In some ways it reflects of my distinction between Active and Inactive (or Passive) entities in a panel (originally based on Natsume's distinction of "positive" vs. "negative" entities). However, my breakdown is superficially "things that move across panels" to "things that don't." Protagonists could fall into either one of those categories given the appropriate sequence.

But... what if there is more than one protagonist? What if a scene shift happens where a new character becomes the lead character — this would just be coded as a consistent "-"?

Mostly though, I am unsure of what is interesting about these results. The visual language in comics features consistent "runs" of protagonist or non-protagonist panels: so what?

The analysis throughout focuses only on linear sequences based largely on Markovian chains, but I think my work has strived to show that sequences of images cannot simply be considered linear sequences. They have hierarchic structures guiding them — which such a binary analysis of the surface elements would be unable to show.

This study is an interesting first attempt at using computational methods to analyze visual language structure — and I love that the research has now begun permeating such extents. Hopefully further studies will bring more interesting results.

ResearchBlogging.orgLaraudogoitia, J. (2008). The comic as a binary language. An hypothesis on comic structure* Journal of Quantitative Linguistics, 15 (2), 111-135 DOI: 10.1080/09296170801961785

[Originally posted 6/2/08]

Tuesday, May 03, 2011

Deaf Comics

I have always wanted to learn sign language, and my labmate, who is an interpreter, recently began teaching me American Sign Language. It's been very fun to finally be learning it, and a great education in non-verbal languages (which I'm obviously interested in!).

She also recently sent me a link to That Deaf Guy, a comic about a deaf man and his family. It's a very smart and well done strip, that has some great humor about signing and deaf culture. My favorite are the "Dos and Don'ts" of relating with deaf people. Go check it out!

And, along those lines... here's an awesome deaf rapper!

Monday, April 25, 2011

New Article: Comics vs. Manga

I recently had a new paper appear in the journal Image[&]Narrative about the difference in the structuring of comic panels in American comics and Japanese manga. Here's the abstract:

A different kind of cultural frame: An analysis of panels in American comics and Japanese manga
Neil Cohn

The growing interest and influence of Japanese manga (“comics”) in America has inspired comparisons between the properties of the two cultures’ graphic systems. Various theories have hinted to the existence of structural variation between these cultures’ books, yet little quantitative data has served to support these claims. This study seeks to provide empirical evidence for these cross-cultural theories by examining 300 panels in each of twelve American and twelve Japanese comic books. It examines 1) how they highlight amounts of information, 2) their depiction of subjective viewpoints, and 3) the angle of view taken by their representations.

Click here to download a pdf.

Additionally, the rest of that issue is devoted to aspects of the visual language of manga, so check out other articles if you're interested. Enjoy!

Wednesday, April 20, 2011

Comics and pictorial metaphor

I recently discovered that the "Semiotics Institute Online" has a few full online courses that are offered related to comics. One of those is a class on metaphor pictorial theory taught by Charles Forceville, who has been looking a lot at comics and other media.

Of particular interest is lecture #7, which directly discusses conceptual metaphors in numerous comics, ranging from European comics to manga, Calvin and Hobbes to political cartoons. It's worth reading for a good primer on some of the more creative aspects of meaning-making in comics.

Forceville has also recently set up a blog about his research. The site has abstracts for many of his papers, and in some cases even has pdf downloads available. Again, of particular interest to readers of this blog may be his papers on comics, mostly concerning his work on metaphor theory as well as a categorization system of visual signs (that I admittedly have mixed feelings about).

Go forth and learn!

Wednesday, April 13, 2011

99 examples of narrative

Matt Madden's 99 Ways to Tell a Story is a great exercise in creativity where he draws the same scene (him stopping work to get food, while talking with his partner) in 99 different unique ways. I've known of the book and its great content for years, but since Matt recently gave me a copy of the book, I wanted to make some comments about it...

One of the great aspects of this book is how overtly it shows the difference between meaning and narrative. The meaning is fairly constant throughout all 99 tellings (its what is being told), but the narrative is totally different (how it is told).

This distinction is important, because it can lead us to ask what makes different narrative styles function? How can different narrative techniques all tell the same basic meaning, and how do those styles change the meaning through their presentation?

These questions are actually the motivating issues to most of my recent research. They also reflect the same types of questions that are asked about the construction of sentences:

what makes different [grammar] function? How can different [sentence patterns] all tell the same basic meaning, and how [does that grammar] change the meaning through their presentation?

Friday, April 08, 2011

Busy Times...

I wish I could blog more these days, but life has been crazy. I'm still playing catch-up from my trip to California earlier in the week (and to Rhode Island the week before), but lots has been going on...

Over the past few weeks I've submitted several articles to journals, have readied another, and am in discussions for a book to be released about my theories of visual narrative sometime in the next two years.

Oh yah, and I am trying to get my dissertation up and going, which involves readying four experiments to be run over the summer.

I've also had an article come out in a journal recently — look for an official announcement soon, once I get my own website set up for it. And, another article should follow in the coming months.

Lots of links and things to come!

Tuesday, March 29, 2011

CNS 2011

I will be speaking next week at the Cognitive Neuroscience Society conference in San Francisco. If you're in the area and coming to the conference, I'll be presenting in Slide Session 7, on Tuesday, April 5, 1:00 - 3:00 pm, Grand Ballroom B:

This is your brain on comics: The impact of structure and meaning on sequential image comprehension

Neil Cohn, Martin Paczynski, Phil Holcomb, Ray Jackendoff, Gina Kuperberg; Tufts University

Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing (1, 2) to examine structure and semantics in processing sequential images. Using Cohn’s (3) model of visual narrative, we compared four types of comic strips: 1) Normal sequences with both structure and meaning, 2) Semantic Only sequences (semantic relationships but no structure), 3) Structural Only sequences (structure but no semantic relationships), 4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantics and structure offer advantages to processing. Experiment 2 measured ERPs to the same target panels. The largest N400 appeared in both Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in Normal sequences. This implies that a combination of narrative structure and semantic relationships can facilitate semantic processing (as reflected by the N400). However, the effects of structure alone may be independent of semantics. Taken together, these findings suggest that sequential image comprehension uses a grammar that extends beyond semantic associations between individual frames. The comprehension of graphic narrative is guided by an interaction between structure and meaning, akin to that between syntax and semantics in language.

Wednesday, March 02, 2011

Comic Scramblers

Busy-ness has been in high gear here in visual language research land. I'm working on submitting several papers for publications over the next few weeks, and am over half the way through writing my dissertation proposal. Plus, my wonderful assistants are preparing our next round of experiments...

Speaking of which, one of my assistants, Suzi Grossman, has applied some of my theories to some artistic work. She's made several "scramblers" which generate comics out of random panels. This site has two of them that are made from panels we're using in our research.

The first generator makes completely random strips, so they should feel like a whole lot of gobbledegook. The second generator actually plays with the structure of the narrative though: while certain pieces remain constant, others with particular traits are exchanged for each other. (Press the first green button for new strips. Press the second to swap out that single panel)

She takes this one step further with this project. Here, the generator draws from several different comic strips, but organizes the panels into a coherent narrative, though it might not make sense! I've actually wanted to do a create project like this for some time, so it's great she's actually done it.

Check them out!

Monday, February 07, 2011

Review: Children Interpret a Comic

This insightful article examines children’s understandings of comic books over time using a Western comic A Gunman in Town!. The study looked at ten children in each of 3rd, 5th, and 8th grade, balanced for gender and race with diverse socio-economic status. They were shown each frame individually and asked its contents following each panel. This might not have hugely hampered the sequential understanding though, since the panels seem largely dominated by text.

All the children recognized broader information: that the book was a Western and that it would end with the villain losing. All the children were concerned with the concept that the story was going to end, showing knowledge of it as a story and that stories have endings. Most of the phenomena showed small jumps and differences in understanding between grades. For instance, in readings of the last panel of the book, a steady increase of children recognized the correct reading order of word balloons (Grade:number of kids – 3rd:2, 5th:4, 8th:7).

Many third graders would skip over reading dialogue, especially when it was heavy in panels. They also will gather most of their reading from stereotypic knowledge, missing important story elements or filling in missed information with further stereotypic knowledge about genre.

Fifth graders pick up far more information than third graders, with explanations seeming less stereotypic – allowing them to anticipate and integrate events more quickly and accurately. Eighth graders “move back and forth between their knowledge of conventional genre structure and the particular story” (46). Fifth graders are more capable of predicting future events from individual panels — each panel implies something about future events. While eighth graders can predict to the end of the story, fifth graders make more short-term predictions about action sequences.

Eighth graders see the story as conventionally ordered by the dictates of the genre. Two strategies were used by eighth graders. When uninterested, they use a “flat” style that perceives and decodes the story as it unfolds bit by bit. A contiguous reading style incorporates the understanding of the genre to expand on the given information with schematic knowledge (unlike with third graders, this isn’t to make up for missed information though).

These results further indicate that the ability to understand sequential images increases with age, and perhaps with exposure/experience.

ResearchBlogging.orgPallenik, M. (1976). A Gunman in Town! Children Interpret a Comic Book Studies in the Anthropology of Visual Communication, 3 (1), 38-51 DOI: 10.1525/var.1976.3.1.38

[Originally posted: 12/24/07]

Thursday, January 27, 2011

NY Times book review... of my parents

Every now and then I'm compelled to plug something friends or relations do, and I'm super excited that my parents' latest book has been reviewed in the New York Times.

They have written and published books about treating eating disorders for almost three decades, and this is the 25th anniversary of their first publication, Bulimia, which was the first book ever published that addressed this disorder for laypeople.

So, head on over and read the review, and please recommend the book if anyone you knows suffers from eating disorders.

Saturday, January 22, 2011

Talk this Friday

For those of you in the Boston area...I will be giving a lecture next Friday, Jan 28th at 4pm in the Tufts University Psychology Department that is open to the public. The name of the talk is "What is 'Visual Language'?: What Comics can tell us about the mind" and it covers an overview of the foundation of my theories of the cognition underlying comics.

4pm, Jan 28th
Tufts Psychology Building
1st Floor Conference Room

Thursday, January 13, 2011

New England Comic Arts in the Classroom Conference

Happy New Year all! I've recently been confirmed as a speaker at the New England Comic Arts in the Classroom Conference, a conference aimed discussing the role of comics in education. The conference is open for registration currently, and will take place on March 26 in Rhode Island. If you are interested in attending or finding out more, follow the link above!

A description of my talk:
What Comics can tell us about the Mind (and vice-versa)

Recent research on comics within the realm of cognitive psychology has hypothesized that the capacity to draw, especially in sequential images, manifests in the mind similar to language, with systematic grammatical rules and patterns. This presentation will explore what it means for sequential images to be understood like a language and how that compares to other domains of expression like verbal language, diagrams, and gestures. We will also explore what the ramifications of such a theory might be on education, development, and learning.

Wednesday, January 05, 2011

It's not always about the brain

This topic arose elsewhere recently and I thought it would be worth bringing up here. As someone who now actually does have at least a little data about how the brain processes comics, I feel compelled to talk about others who do as well...

I've noticed several recent articles that appeal to the "cognition" of comics or the "neuroscience" of comics, particularly in popular writings. However, most of these discussions have nothing to do with the brain's processes. As a caveat to anyone who might do this, I'm inclined to repost part an old blog entry that's related:
It is easy to be enticed by the desire to discuss the brain. After all, it is the hidden key to understanding human activity, and I can see how mentioning it lends a feeling of legitimacy to talks of "narrative art." However, in most discussions (like here), it is largely irrelevant. "Word, images, and writing" can adequately be described and interestingly discussed as human behavior without invoking vague pop-psychological discussions of the brain...

It is very hard to make claims about neurological activity (like that "narrative art" involves right or left brain activity and/or their interactions) without some sort of experimentation. Hell, it's hard to make conclusive claims about the brain even with experimentation! (...which is partially what makes it so intriguing to study).

My point overall is this: as cool and interesting as it is, not all arguments need to be tied to the brain and cognition. And, in fact, some arguments are made weaker by doing so, since appealing to neuroscience is unnecessary at best and hand-waving at worst.

Figure out what your point is and talk about it. I'm guessing it actually has little to do with neuroscience directly.