Sunday, March 30, 2008

Thoughts on Language Evolution

In my TA class this semester we've just entered talking about Language Evolution, and combined with the recent discovery of this blog the topic has been on my mind a bit lately. Some general thoughts on reframing the overall discussion...

Some theories of language evolution postulate that ‘gestural language’ evolved prior to verbal language. While I am in support of the multimodal sentiment, parts of this rub me the wrong way.

First off, humans currently use both modalities concurrently in expression, which is offered as part of the evidence for its potential importance in evolutionary contexts. While the verbal form in most people uses more complex structure, both forms are in use at once in co-speech gesture. Why would it make sense for only one modality to develop dominantly then transfer into another? Should we perhaps be thinking in terms of concurrent development for concurrent usage?

Along these lines, humans also have the capacity to draw. While our primate cousins do have gestures and vocalizations for conceptual expression, none seem to manipulate the world for conceptual intent, which is at the heart of drawing. Yet, this is never mentioned alongside the discussions of the other modalities (though not surprisingly).

Also, I think it is largely a misnomer to say “gestural language” evolved first. As a cover phrase, I think it obfuscates the issues involved — namely, that there are various mental structures that contribute to a behavior of manual (verbal and visual) expressions.

This is even more problematic when various people use the word “language” in different ways — some refer to a communicative system, some to grammar, some to conceptual expression. Really, if the discussion is about the evolution of what we now know of as “language”, it may indeed be inappropriate to talk about its historical states as if they're the same thing as we have now.

It would be more useful to discuss the development and evolution of these structures (syntax, semantics, etc) than to talk about the capacity as if it were a whole thing. That is, we should be talking less about the evolution of “Language” and more about the development of the interacting cognitive structures that end up contributing to language as we know it in the modern context.

Monday, March 24, 2008

Art, Language, and "Cognitive Equivalence"

When I usually speak about the Art versus Language Perspectives, I usually couch it in a view that there are "different potential ways our society treats graphic images." As I just realized, stating it in this way maybe obscures the true intent of the distinctions.

Really, this is a hypothesis about cognition.

At the heart of my theory of visual language is the observation that we have three modalities by which we can convey our concepts: Sound, Body motions (faces/hands), and creating graphic images. That's it. To push it further, the theory is that when these modalities take on structured sequences governed by a system of constraints (a grammar), it becomes a type of language: speech, sign language, or visual language.

The "Language Perspective" assumes that all systems of conceptual expression work in similar ways — what we can maybe call "cognitive modality equivalence" or some such. Under such a view, we would expect for the graphic modality (drawings) to operate under the same principles as the verbal and manual domains.

If you look at the ways that speech and gestures (and sign language) grow developmentally in children and are used and treated in society, you see certain patterns — conventionality, imitation, communality, etc. While many of those patterns do emerge in the graphic domain, they appear "dampened", are dismissed, or just aren't recognized as such.

So, the question becomes raised: "Why don't you see these things fully in the graphic modality?" and/or "why don't we know them when we see them?"

The offered explanation is the Art Perspective — a cultural force that suppresses the patterns that would normally emerge from any other modality of conceptual expression. With polar opposite emphases, the Art Perspective works to dampen the "usual" course of development and treatment for graphic images.

So, given this, a new set of questions can be asked about this underlying "cognitive modality equivalence": What are the trends that a conceptually expressive system shows in development and society? How do modalities differ? Do these trends reflect broader cognitive processes than just conceptually expressing systems?

Sunday, March 16, 2008

Closure's assumptions

Patric continues his defining of "comics" with a discussion of "closure." I've talked before about the problems with the idea of closure, but it strikes me that there are a few underlying issues that people run into when addressing these issues:

1. They assume that time passes between panels, despite there being no evidence that each panel represents a "moment in time." With this assumption in place, it forces people to assume that some "moment" also lies between the panels, when no hidden moment may exist. I wrote my essay "Time Frames...Or Not" about various reasons why this assumption isn't true.

Even McCloud bungles this. While in one place he tries to say that "panels=moments" because "time=space", in his own transitions he includes three that have nothing to do with time at all! (Subject, Aspect, and Non-Sequitur transitions). For the adamant, what are the moments and what are the transitions in this "comic"?

2. People are just looking at the relationships of two juxtaposed panels. Most stabs at sequential meaning, like Patric's or Derik's, have just talked about two-panel pairs. But, rarely are sequences confined to two panels.

Just because we experience reading sequences of images linearly doesn't mean that is how we understand them. In most cases, we can easily acknowledge that whole sequences mean something beyond just paired panels. Looking beyond the scope of immediate panel relations quickly forces a rethinking of the accuracy of a view about closure/transitions.

Here are a few illustrative exercises that people can do to think more about these issues (and are things I did when first getting into this seriously):

1. Actually try to catalogue the "transitions" in a full comic รก la McCloud's counting. Note any problems in the categories and where descriptions become more difficult.

2. Take comic pages/strips and sketch out the different relationships of every panel to each other. Which panels need connections, which don't? What do the relationships tell you?

If anyone actually does this, I'd love to hear about their results. In the meantime, if people are curious about my alternatives to closure, I recommend watching this video.

Monday, March 10, 2008

Double Standards in Style

I had an interesting realization the other day about the way people judge the quality of realistic versus cartoony drawing styles. It seems to me that the more someone tries to maintain a realistic style, the more harshly criticized they will be when they don't "fully achieve" it. Cartoony styles get no critique like this.

As I've done before, perhaps Rob Liefeld will be a good example. Liefeld is often criticized for his unrealistic body proportions and suspect anatomical correctness — despite being proficient in his craft. However, I've never heard of Matt Groening criticized as having a poor understanding of anatomy for the Simpson's only having four fingers, or that practically no one has chins.

It seems to me that with cartoony styles, we accept that drawings are more of a representation of a concept than a re-creation of "reality." The more realistic the style becomes, the less accepting people are of this, yet it still remains true: The capacity to draw is for representing thoughts visually.

Wednesday, March 05, 2008

America vs. Japan: Brains and Comic/Manga Panels

Via the TCJ message board, Nathan has pointed to an article in the Boston Globe that discusses the differences in brain activation between "Eastern and Western" perceptual processing. The study claims that "Westerners tend to focus on central objects more than on their surroundings" while Easterners "tend to focus more on the context as well as the object." From the article:
To use a camera analogy, "the Americans are more zoom and the East Asians are more panoramic," said Dr. Denise Park of the Center for Brain Health at the University of Texas in Dallas. "The Easterner probably sees more, and the Westerner probably sees less, but in more detail."

"Literally, our data suggest that people see different elements of pictures," Park said. "If you're looking at an elephant in the jungle, the Westerner will focus on the elephant and the Easterner is going to be more thinking about the jungle scene that has the elephant in it."

In a way, these findings are supportive of McCloud's claims that manga use more "wandering eye" type of panel "transitions." The evidence from my own more formal study comparing panels from Japanese and American comics (in my paper Cross-Cultural Space) seems to support this conclusion... somewhat.

My study found that American comics by far used more comic panels that featured a whole scene ("Macros"), while Japanese manga used equal amounts of panels with whole scenes and individual characters ("Monos"). Manga also used a great deal more "Micro" panels, which feature a "zoom."**

These results would seem to support a view that Japanese panels allow a focus on the broader environment, since they are breaking up the single environment into smaller parts. However those smaller parts are giving focus to the smaller parts instead of to the larger whole. So, in a way, manga panels are getting both the environment and the detail of the objects.

Unfortunately, my coding in this study was a little deficient, since at the time I lacked an "Amorphic" category that contains purely environmental information. These panels were coded as Micros at the time, but really should be their own category. On the plus side, I now have a larger and more diverse sample of comics to code and a richer coding scheme, I just need to get the peoplepower to do it (read: undergrad research assistants).

Update: An additional thought I just had related to this is the extant to which these claims are generalizable into two categories of East vs. West. At least regarding the graphic form, will we find that American books are the same/different as various European books? Can Japanese manga really be lumped in with Chinese, Korean, and other Asian comics' structure? Perhaps we'll find that there's a lot more diversity out there than we suspect...

**(The graph above shows a reanalysis of these numbers, getting rid of two American books that had "high manga influence" — the difference is slight but significant. Check the paper for initial interpretation/numbers)

Monday, March 03, 2008

The Art of Visual Language

As much as I stress how the Art and Language perspectives/paradigms of viewing graphic communication are opposed to each other, I do think that they can be reconciled. Just to recap, I believe that a cultural force, what I call the "Art Perspective" suppresses the visual-graphic form of expression, which is closer to a "Language Perspective." Some of the things that differ in these paradigms' emphasizes are:

Art Perspective
• Individuality and Innovation
• Imitation is bad
• Iconicity = just perception
• Cross-cultural Universality
• Innate talent or skill
• Etc.

... Versus...

Language Perspective
• Communally used signs
• Imitation is central
• Patterned graphic regularities
• Diversity requiring fluency
• Innate potential for acquired system
• Etc.

What I've argued for mainly is that at the base understanding, the visual-graphic modality should not be thought of as "Art." Rather, it should simply be considered a mode of conceptual expression, whereby "Art" is a socio-cultural designation that may or may not be applied to its usage.

So, given this perspective, the Art viewpoint can easily be applied to usage of this visual language at least on some levels.

Individuality and innovation can be applied to usage and not form. For instance, instead of emphasizing that people should all draw in unique and different ways, it would be easy for people to all use a common style whereby they create novel expressions with those signs. On this level, imitation could still be "bad" — while imitation to establish the skill set for drawing is necessary.

Take manga for instance. Many (certainly not all) manga are drawn in a conventionalized style in which authors have individualistic voices, yet largely share a common visual vocabulary. But, they can then have potential to create novel stories and expressions out of that conventionalized vocab. (This doesn't mean that they do it per se, there is still quite a lot of derivative stories in manga).

The "Talent" issue can also come out here. Instead of judging people on how "good they are at drawing", given an acknowledgment of graphic fluency, talent is recognized for what people do with their drawing ability, just like we recognize that good writers make use of their common vocab by using it in inspiring ways.

In all cases, the emphasis here is that the Art perspective is not necessarily applied to the image-making itself (or the development of that ability), but rather it's about the usage of that capacity. The Language perspective holds true at the base level of identifying the capacity for (sequential) image-making, while the Art perspective is applied interpretively to its usage.