Wednesday, November 28, 2012

Discover Magazine article

I'm proud to say that I'm featured in an article in this month's Discover Magazine! The article was written by the excellent Carl Zimmer, who good-naturedly let me run him through my experiment for the sake of the article. I'd actually read articles by Carl for many years, so it was fun to interact with him  for the interview.

I should note a slight correction to the the reported results of my study. While the difference between the Scrambled and Narrative Structure Only sequences did show a "left anterior negativity" (correlated with syntax), the difference between the amplitudes of those sequences and normal ones showed a different waveform, called the N400 (correlated with semantics). So…

Normal vs. Structural Only = N400
Normal vs. Scrambled = N400
Structural Only vs. Scrambled = Left Anterior Negativity

You can read the original article here (pdf), or a short, "comic" version here (pdf).

Overall though, Carl did a great job describing my study and this type of research. I'm very humbled to receive the attention. Go read!

Tuesday, November 20, 2012

Sequence vs. Singles in "visual language"

In my last post, I addressed the basic idea for a "visual language" as being a sequence of meaningful images guided by a system of constraints (i.e., a grammar). In the comments, I was asked a very good question:

Why is a sequence necessary for the graphic form to be considered "language"?

There are two main reasons for this, both which relate to the analogy with verbal and sign languages. As I said in that post, my notion of "language" in "visual language" is not metaphorical, but rather based on conceptions from the linguistic sciences.

The first reasons is structural. Languages are made up of three primary components:

1. The conceptual structure of meaning in the mind
2. A sensory modality they can be expressed in (i.e., sound, body motions, graphics)
3. A "grammar" that guides and constrains the sequential expressions of meaning

In the verbal form, the main grammar is syntactic structure, which allows us to sequentially order words and other expressions into coherent sentences. However, technically all of these components (meaning, modality, grammar) are built of rule-systems that constrain them. The phonological system that guides our production of sounds also is constrained by rules. This is why English cannot have words that start with the sound combination "tf" or why the "c" in elastic goes from a "k" sound to an "s" sound in elasticity. These are rules guiding the modality itself.

The analogy for the graphic form holds these same functions. The primary "grammar" guiding images is a "narrative grammar" which guides the presentation of meaning in coherent sequences. Both syntax and narrative function in the same general way: to present meaning in a coherent sequence. They also share methods of doing this, such as chunking units into groupings, making connections between distant units, and embedding groupings inside each other.

However, single images also have a constraining system which is analogous to phonology. You could call it "photology" or "graphology" perhaps. This system similarly constrains the modality itself. This is why certain junctions of lines are awkward, like when you want to show occlusion (one thing in front of another), but instead of using a "T" shaped junction of lines, you use a "Y" or "+" shaped junction.

So, structurally, single images are guided by a rule system, but that system is closer to that of phonology than syntax.

The second reason for sequence being important comes from analogies with development.

By and large, when people are not exposed to a language within the right time period of life, they won't learn language. They seem to be able to still acquire a limited set of vocabulary (i.e., words) but the most problematic component is the syntax.

Even when people are able to learn a spoken language, but never learn sign language, they can still all gesture. The manual modality doesn't disappear as a way to create meaning—it just functions using single expressions without a grammatical sequence.

This same trend is true of drawing and sequential images. Most people cannot draw a coherent narrative sequence. However, they can all use the drawing system ("photology") to create meaningful single images (albeit rudimentary ones if they haven't fluently developed the vocabulary of the drawing system either).

So, the analogy then holds that single images are to visual language what gestures are to sign languages. One type uses a modality for single novel expressions (single images/gestures) while the other uses complete grammars in sequences of expressions (sign language/visual language). The evidence comes because even a rudimentary form of the simpler expressions (single images/gestures) is maintained even if the full grammar is not developed.

Incidentally, quite a lot of my discussion about the structure of single images and the development of the drawing system is available in my recently published paper, "Explaining 'I can't draw'" available as a pdf here.

Tuesday, November 13, 2012

Revisiting "visual language"

I've now had this website for over 10 years, and have been blogging for almost 6 years, so it may be worth revisiting the fundamental ideas of my research over the next few posts. Hopefully by this time next year my book, The Visual Language of Comics, will be out and describing these ideas in even more detail. Until then...

Let's start with the obvious: What is "visual language"?

There are several ways that the term "visual language" can be used. Sometimes it is used to talk about general visual information or visual culture. It might be used as a broad term for visual culture, or for any combination of text and images. Some people use it to describe creative ways to use writing in pictorial ways.

None of of these are what I mean by "visual language."

This isn't necessarily a bad thing. These and other applications of the term use "language" in a very metaphorical sense, usually by extension to mean "communication."

My meaning of "language" is actually very literal, based on the scientific definitions of language. By extension, my definition of "visual language" is also very specific.

So, what do I mean by "visual language"?

Human beings as a species can only convey our thoughts in three ways: we can 1) create sounds with our mouths, 2) move our bodies (especially hands and faces), and 3) draw things. That's it.

When any of these channels is put into a sequence, such that some sequences are good and others are bad, then the result is a "language." Thus, sequential sounds (words) become spoken languages, sequential body movements become sign languages (as opposed to gestures) and sequential images literally become visual languages.

Given this, individual images are similar to single expressions (which have their own rich structure), while sequential images form a visual language.

So, what is writing? Writing is the learned importation of the spoken form into the visual form (essentially a learned synesthesia). This is not natural, which is why it's so hard to learn to read and write, and why most of the world's languages use no writing systems.

By contrast, the ability to draw sequential images is a natural ability that is accessible to anyone who receives the proper exposure and practice at it.

Given all that, now what about "comics"? Well, comics are the place that we predominantly find these visual languages used. Just like novels are written in English, comics in America are written in American Visual Language. Or, manga in Japan are written in Japanese Visual Language.

And, of course, comics are not just written in the visual language of sequential images, they also use written language. So, technically, comics use two languages that combine to make a larger whole of communication. This is actually similar to the way we communicate generally. We constantly combine modalities: we gesture when we speak, text and image often come together, etc.

From this basic idea, that sequential images literally create a natural visual modality of language,  innumerable other questions emerge about the nature of graphic communication, it's cognition, and how it can be used in society.