Thursday, December 20, 2018

2018: My publications in review

The last few years I've closed out the year by summarizing all of my papers that came out (2016, 2017), and so this year I'm doing the same. It's been a diverse year of papers, with some theoretical papers, a few brainwave papers carried out by colleagues, and a corpus study. So, here are the papers that I published in 2018...

The cultural pages of comics (PDF) - This paper coauthored with my student assistants followed up our analysis of page layouts in superhero comics by comparing page layouts in 60 comics, 10 each from US superhero comics, US Indy comics, Japanese shonen manga, Hong Kong manhua, French bande desinée, and Swedish comics. Overall, we found that cultures differ in their page layout features in patterned and systematic ways. For example, layouts in Asian comics use more vertical segments, while those from Europe and US Indy comics use more staggering of panels within horizontal rows.

In defense of a “grammar” in the visual language of comics (PDF) - This theoretical paper reviewed my theory of narrative structure, and defended it against critiques that sequential image comprehension requires only meaningful connections between panels. I review and compare the theories, and lay out arguments for why a narrative structure is both necessary and supported by the experimental evidence. I also take the hard line that any proposal for how visual narrative sequences are understood must account for the cognitive results in experimentation.

Combinatorial morphology in visual languages (PDF) - In this chapter from the recent book The Construction of Words: Advances in Construction Morphology, I try to formalize the linguistic structure of the morphology ("symbology") of visual representations like hearts or lightbulbs above the head, motion lines, and impact stars. It discusses both how these forms use systematic strategies to combine elements, and the ways they derive meaning through symbolic and metaphorical techniques.

Listening beyond seeing (PDF) - My coauthor Mirella Manfredi carried out this cool study which showed people comics, and at the critical panel also played sounds to people. The panel showed an action, while either playing people a spoken onomatopoeia that matched or mismatched the action, or an actual sound effect that matched/mismatched the action. We measured people's brainwaves, and found that their processing of these multimodal meanings partially overlapped, but partially did not. Brainwaves to words and sounds differed at the start of their processing, but in later parts of the processing seemed to not differ, implying some sort of integrative process.

Visual Language Theory and the scientific study of comics (PDF) - This chapter appeared in the recent book Empirical Comics Research, which has a wide survey of studies using empirical methods (corpus, computational, cognitive) to study comics. My paper provides a review of my Visual Language Theory, and its structures of vocabulary, layout, and narrative structure. I describe how theories of their structure combines with corpus analysis and psychological experimentation to give us a converging view of how visual languages in comics are built. I think it's a relatively decent introductory paper for people who are unfamiliar with my theories.

Are emoji a poor substitute for words? (PDF, Poster) - Our conference paper from the 2018 Meeting of the Cognitive Science Society looked at how people process sentences when emoji are substituted for words. We found that people view emoji slower than words in sentences, but even slower when the emoji mismatches the part of speech (ex. a "noun-ish" emoji in verb position). When people read the next word after seeing a congruous emoji, they process it just as easily as seeing an all text sentence, but words after incongruous emoji are still read slower. This suggests that congruous emoji substituted for words can readily be integrated into the syntax of sentences. We also compared logos and emoji substituted in text, and found they didn't differ in their processing.

Visual and linguistic narrative comprehension in autism spectrum disorders (PDF) - My first paper with my colleague Emily Coderre compares the brainwaves of neurotypical individuals with individuals with autism while they comprehended both verbal and visual narratives. People have often claimed that autistic individuals do better with visual materials, but we show similar processing deficits for both verbal and visual materials, hinting at a more general issue processing meaning across modalities. This is the first of my papers on autism and visual narratives with Emily, and we've got lots more on tap coming soon.

Workshop: How we make and understand drawings - Finally, not a publication, but back in April I gave two workshops at the University of Connecticut with philosopher Gabe Greenberg where we examine the structure and meaning of individual and sequential images. My portion (first day) examines how drawings are structured and how people learn to draw, which starts midway (02:18:15) through this video:

On the second day, my portion reviewed my findings about how visual narratives are processed, particularly the combination of narrative structure and meaning. I then presented my multimodal model of language and cognition. That's in the second half of this video (02:04:20), which unfortunately has less good sound:

Forecasting ahead to next year, I can already say that it's going to be a big year. I have a special issue of a journal that I'm editing that has some great looking papers. I also have two big review papers that should be coming out, one on processing and one on "fluency" of sequential images. Plus, we've now run five (!) brainwave studies in my operational EEG lab here in Tilburg, all of which are being written up. So, here's looking forward to a good 2019...

These and all my papers are available on my website here.

Monday, December 17, 2018

Review: Metaphoricity of Conventionalized Diagetic Images in Comics

Michał Szawerna's recent book Metaphoricity of Conventionalized Diegetic Images in Comics: A Study in Multimodal Cognitive Linguistics analyzes a variety of structural aspects of the visual languages of comics by taking a deep dive into Peircean semiotics and cognitive linguistics, particularly conceptual metaphor theory, and cognitive grammar. The book seems to have flown largely under the radar of most discussions of comics theory, but it is interesting in several regards.

The book opens with an analysis of the history of scholarship on comics, emphasizing the structuralist and linguistic analyses. Included in this is a discussion of Polish research, which I had not previously seen discussed in other publications. It also extensively covers the semiotic theories of C.S. Peirce and the developments of conceptual metaphor theory over the past 30 years.

The substantive chapters then each delve into a different aspect of the structure of comics. This starts with a chapter on the abstract properties of panels and how they convey time across sequences, then progresses to a discussion of depictions of motion (motion lines, polymorphic panels).  Chapters then discuss the depictions of sound (balloons), and "mental experiences" (like thought bubbles, upfixes). A concluding chapter then summarizes the overall arguments.

The book throughout contains several insightful examples and analyses, and at the least makes one consider the complexity of various visual conventions. For example, the chapter on motion discusses what I've called "polymorphic" representations, where a single panel shows a character repeated in an action to imply motion. Here Szawerna observes that this overall pattern extends beyond motion, and can also depict transformations, like a werewolf's shift from a man to wolf-man. I don't think I've seen this representation discussed in any other paper, and it's  nice observation of its similarities to other polymorphic panels.

Other observations seem a little overly strong. For example, in the chapter on comic panels, Szawerna takes on the strong McCloudian position that the width of panels has a direct correspondence to time duration. He also claims that images in sequence are directly mapping to a timeline of episodic events (a space = time metaphor), even comparing comics to the grid pattern of days on a calendar. I've long pointed out problems with this view, and support against it has been provided by several experiments.

This relates to my first critique of the book. Though the book has many good insights, it ultimatley feels like a case of “if all you have is a hammer, then everything looks like a nail.” That is, the metaphorical interpretations run so rampant throughout that no alternative interpretations are offered nor considered. I don't disagree with metaphorical interpretations of various conventions, but it seems a metaphorical interpretation should be a "last resort" if a simpler explanation is possible. For example, experimentation of motion lines has implied their understanding is not metaphorical or based on our perception of moving objects, but driven largely by conventionalization.

Also, while the work is clearly well-researched, at times references seem selective or miss important arguments. For example, in the introductory chapter, Szawerna critiques my notion of visual language on the basis of Hockett’s design features for language, claiming that visual languages cannot be languages because they do not exhibit thing like duality of patterning or arbitrariness. However, these issues are addressed in the second chapter of my book, which is cited, and perhaps more importantly, does not acknowledge that those features do not hold up for sign languages, nor are they even consistent descriptors of spoken languages.

My second main critique of the book relates to cognition. Mostly the book seeks describe what is happening in the visual language of comics, often in very intense details. But, these often amount to just giving labels to things, falling short of explaining the mechanisms and cognitive processes involved in these representations. Granted, description is important too, but I would have hoped for more of a balance.

More concerning is the repeated invocation for the “psychological reality” of the argued analyses, despite no evidence being provided for such interpretations. There are no theoretical diagnostic tests, nor is any empirical literature discussed, even though there has been relevant psychological experiments about many of the issues under analysis.  Claims of "psychological reality" need to engage the actual experimental cognitive literature, as should any theoretical claims about how "comics work."

For example, the experimental literature would especially be useful to examine Szawerna's claim that people transparently understand images and conventions in visual languages (which he attributes to Miodrag). The empirical literate actually shows cultural differences for many conventions that occur in comics (and even basic drawings). Also, developmental psychology has shown trajectories for learning to understand basic images, image sequences, and morphemes like motion lines and carriers. Szawerna uses the assumption of transparency to ground claims of metaphoric knowledge motivated by universal and embodied understanding, but the literature does not seem to support this (although, non-transparency does not rule out a metaphoric interpretation).

Finally, it should be noted that stylistically this book is not an easy read, particularly for those who don't often read research on linguistics. It is often weighed heavily by jargon and exceedingly long sentences. Some serious copyediting could beneficially cut at least a third of the book's 490 page length. This would have been useful, as I fear that sometimes the book’s insights are buried beneath the prose.

Criticisms aside, the book seems like it would be important for scholars to engage if they are interested in the understanding of these elements of visual vocabulary and/or visual metaphor. In addition, this book seems to be a landmark in the study of the visual language of comics for what it does. It is the first, to my knowledge, to devote a book extensively to rigorously analyzing just a few structural features of the visual domain. Such depth of analysis is indicative of the growing seriousness and sophistication of the linguistic and cognitive approach to visual languages, hopefully making Szawerna's book a harbinger of further works to come.

Szawerna, Michał. 2017. Metaphoricity of Conventionalized Diegetic Images in Comics: A Study in Multimodal Cognitive Linguistics, Łódź Studies in Language 54: Peter Lang Publishing.