Sunday, September 18, 2016

New paper: Meaning above the head

I'm happy to announce that our new paper, "Meaning above the head" is now published in the Journal of Cognitive Psychology! This one explores the structure of "upfixes" which are the class of visual signs that float above character's heads, like lightbulbs or hearts.

In my book, The Visual Language of Comics, I made a few hypotheses about these elements. First, I argued that they were bound by a few constraints: 1) they are typically above the head, and are weird when moved to the side. 2) the upfix has a particular "agreement" relationship with the face (e.g., storm clouds go with a sad face, but are weird with a happy face). Also, I argued that upfixes are an abstract class, meaning they can easily allow for new ones, though they won't be quite as comprehensible as conventional ones (as in the image below).

With these hypotheses stated, my enterprising student Beena Murthy set out to test these ideas as part of a experiment she ran for a class project (many of the projects from that class are now published). We were then joined by my collaborator Tom Foulsham who aided us in testing additional questions in a second experiment (which you may have taken online!).

Lo and behold, most all of my hypotheses appear to be borne out! Overall, this means that upfixes use particular constraints in their construction, and allow for the creation of new, novel signs! We now plan to follow up these experiments with several more.

Check out the paper, which is available on my Downloadable Papers page, or directly here: PDF.

AND... don't forget that you can also get awesome t-shirts with both the normal and unconventional upfixes. The shirt designs (which are the in this post images) actually feature our stimuli from the experiments!


Abstract:

“Upfixes” are “visual morphemes” originating in comics where an element floats above a character’s head (ex. lightbulbs or gears). We posited that, similar to constructional lexical schemas in language, upfixes use an abstract schema stored in memory, which constrains upfixes to locations above the head and requires them to “agree” with their accompanying facial expressions. We asked participants to rate and interpret both conventional and unconventional upfixes that either matched or mismatched their facial expression (Experiment 1) and/or were placed either above or beside the head (Experiment 2). Interpretations and ratings of conventionality and face–upfix matching (Experiment 1) along with overall comprehensibility (Experiment 2) suggested that both constraints operated on upfix understanding. Because these constraints modulated both conventional and unconventional upfixes, these findings support that an abstract schema stored in long-term memory allows for generalisations beyond memorised individual items.

Full reference:

Cohn, Neil, Beena Murthy, and Tom Foulsham. (2016). Meaning above the head: combinatorial constraints on the visual vocabulary of comics. Journal of Cognitive Psychology. 28(5): 559-574.

5 comments:

Marc van Lburg said...

Hello Neil,

Here's a slightly alternative interpretation,
I have suggested to make a difference between feelings and emotions.
The suffix represents a feeling and the face expresses an emotion.
This allows the character to feel good but be sad about it.
like when you're crying, but that feels kind of good, or you're laughing in the company of others, but you are feeling sad inside.
http://the-euclideanfly.blogspot.nl/2013/03/emanata-2.html
I'm not in a position to do experiments like yours but I imagine that in a test it might still be interpreted as 'weird' just not 'weird' in a graphical sense (?)

grts
marc

Neil Cohn said...

Thanks for the comment Marc. I'm not sure what you mean by "weird" in a "graphical sense". Certainly faces show emotions. But, upfixes convey emotions (happiness), feelings (love), or general states (thinking), but most often only in combination with a face. Only rarely do they use symbols that can stand alone and retain their meaning (hearts).

Our experiments do show that mismatch faces are considered as unusual, and people seem less able to integrate the face and upfix into a whole conception. They only tend to keep them separated (in their explanations) when they mismatch, which shows that something isn't quite right about them. We have some upcoming experiments that plan to look at this more too.

Marc van Lburg said...

hello Neil,
thanks for the reply,
Sorry, you're right, with 'weird' I was referring to the response following the mismatch between face and upfix.
With graphical sense I mean that the mismatch might call upon the same schema that is called upon when encountering any situation with a mismatch between face and feeling.
I was also thinking about the idea that the kind of upfixes that show representations like the trees and rainbows in your examples may be less in accordance with the reality of the brain that has no representations. The brain does however know wave-like events and electric disturbances like in epilepsy. Therefore these would be interpreted more automatically and effortlessly.
I'm definately interested in those upcoming experiments.
marc

Marc van Lburg said...

hello Neil,
thanks for the reply,
Sorry for the confusion, you're right, with 'weird' I was referring to the response following the mismatch between face and upfix. (could that response be represented as @#*!?! btw?)
With graphical sense I mean that the mismatch might call upon the same schema that is called upon when encountering any situation with a mismatch between face and feeling.
I was also thinking about the idea that the kind of upfixes that show representations like the trees and rainbows in your examples may be less in accordance with the reality of the brain that has no representations. The brain does however know wave-like events and electric disturbances like in epilepsy. Therefore things like spirals or wavy lines might be interpreted more automatically and effortlessly.
I'm definately interested in those upcoming experiments.
marc

Neil Cohn said...

In our study (via the link), the "unconventional" ones were rated consistently lower than the conventional ones, including mismatches. You're talking about the unconventional ones with your examples here. I'd agree that upfixes that have some semantic association might be easier than those without, but on the whole, all unconventional ones are worse than conventional ones. Trees and rainbows have semantic associations just the same as waves and electricity do.