Saturday, December 21, 2019

2019: My publications in review

It's now become an annual tradition for me to summarize my publications from the past year (2016, 2017, 2018). Well, 2019 has been an exciting year of papers for me, mostly because almost all of them are review papers—many of which I'd been working on for years! So, here's what came out in 2019...

Your brain on comics (blog, open access paper) - This paper presents a model of the mechanisms the brain uses to processes a sequence of narrative images, informed by my studies on (neuro)cognition over the past 10 years. It proposes that there are two levels of representation involved in comprehension—semantics and narrative structure—and thus  proposes the Parallel Interfacing Narrative-Semantics (PINS) Model. These neurocognitive mechanisms are then compared with those used in other domains, such as language processing.

Being explicit about the implicit (blog, open access paper) - Inference is often discussed about how comics communicate, but scholarship about it often remains very general. This paper categorizes specific patterns used in visual narratives to evoke inferences in a reader. Because such techniques are used (in the image to the right), it shows that inferences don't just happen by chance, but are directed in specific ways by an author's choices and narrative patterns.

Visual narratives and the mind (blog, pdf preprint) - This review paper explores the stages of processing involved with comprehending a sequence of images. It then explores the degree to which these mechanisms might overlap with those from other domains, such as language, and explores the stages of development that kids go through in learning to comprehend visual narratives. It's a bit less technical than the "Your brain on comics" article, making it good for a wider audience.

The neurophysiology of event processing in language and visual events (blog, pdf paper) - This book chapter explores what neurocognitive research tells us about how we comprehend events. Specifically, it notes the similarities in neurocognitive mechanisms used to comprehend language and perceived visual events, and those drawn in visual narratives like comics.

Structural complexity in visual narratives (blog, pdf preprint) - This chapter in the book Narrative Complexity explores questions of complexity regarding the structure of narrative patterns. I explore how various narrative schema combine to create complex patterns (image to the right), and then do a cross-cultural analysis of those patterns to show that they differ in how much they are used between Western and Asian comics. I then close with a review of the neurocognition of visual narratives.

Visual narrative comprehension: Universal or not? (blog, open access paper) - This review paper asks to what degree visual narrative sequences are universally transparent to understand. I review cross-cultural work on people who have difficulty comprehending sequences of images, developmental work on when children start comprehending image sequences, and clinical work on autism, developmental language disorder, and aphasia examining the limitations of their comprehension. These results all show that understanding sequential images requires a fluency acquired from exposure to comics and drawn visual narratives.


Besides these papers, I was ecstatic to learn I'd received an ERC Starting Grant along with some other funding for projects related to visual narratives and autism (with Emily Coderre and co.) and developmental language disorder (with Annika Anderson and co.). So, here's looking forward to an exciting 2020!

Wednesday, December 18, 2019

New paper: Visual narrative comprehension: Universal or not?

My latest paper has now been published in Psychonomic Bulletin and Review entitled "Visual Narrative Comprehension: Universal or not?" This paper explores to what degree sequences of images are universally transparent, and questions this basic assumption that everyone can easily understand a sequence of images with no learning or decoding.

This has been a pervasive assumption amongst many scholars, particularly ones who protest my notion of a visual language, and yet I continually found evidence against this view. I had several researchers tell me of experiences they had where participants in their cross-cultural research could not understand a sequence of images. Then, in developmental research with kids, it became apparent that they didn't understand image sequences until around age 4-6.

This research was troubling because many researchers were using visual narratives in their experiments as stimuli, without questioning how they worked or whether they were understood. This was especially true in research with young kids, where visual narratives were used as stimuli to study the developmental trajectory of different abilities, yet the kids were often too young to understand the stimuli themselves!

Another place where visual narratives were used as stimuli was in clinical research. Visual narratives are frequent stimuli for neurodivergent populations like individuals with autism or developmental language disorder. They are also used in studies with people who have brain damage that affects language, in aphasia.

So, I decided to research all of these topics, and found that all of these contexts have results where people do not comprehend a sequence of images in a "universal" or transparent way. This paper is the result of over five years of research on this topic, and it actually left out quite a lot! (it will thus be the topic of my next book, out next year)

You can find my open access paper online here.

Abstract:

Visual narratives of sequential images – as found in comics, picture stories, and storyboards – are often thought to provide a fairly universal and transparent message that requires minimal learning to decode. This perceived transparency has led to frequent use of sequential images as experimental stimuli in the cognitive and psychological sciences to explore a wide range of topics. In addition, it underlines efforts to use visual narratives in science and health communication and as educational materials in both classroom settings and across developmental, clinical, and non-literate populations. Yet, combined with recent studies from the linguistic and cognitive sciences, decades of research suggest that visual narratives involve greater complexity and decoding than widely assumed. This review synthesizes observations from cross-cultural and developmental research on the comprehension and creation of visual narrative sequences, as well as findings from clinical psychology (e.g., autism, developmental language disorder, aphasia). Altogether, this work suggests that understanding the visual languages found in comics and visual narratives requires a fluency that is contingent on exposure and practice with a graphic system.

Full reference (in Early View):

Cohn, Neil. 2019. "Visual narrative comprehension: universal or not?" Psychonomic Bulletin & Review. 1-20. doi: 10.3758/s13423-019-01670-1.