Tuesday, November 27, 2012

Action-Based Language: A theory of language acquisition, comprehension, and production

This is the paper by Glenberg and Gallese.  How could not skip ahead to this one?!  I mean, the title does seem to imply that it will provide the answer to how language works!  So let's dig in.

Here's a quote:

our understanding of linguistic expressions is not solely an epistemic attitude; it is first and foremost a pragmatic attitude directed toward action.
So all of language reduces fundamentally to the action system?

One caveat is important. Whereas we focus on the relation between language and action, we do not claim that all language phenomena can be accommodated by action systems. Even within an embodied approach to language, there is strong evidence for contributions to language comprehension by perceptual systems 

Whew!  I was going to have to quote Pillsbury again:

A reader of some of the texts lately published would be inclined to believe that there was nothing in consciousness but movement, and that the presence of sense organs, or of sensory and associatory tracts in the cortex was at the least a mistake on the part of the Creator” (Pillsbury, 1911) (p. 83)
On page 906 we get to learn about the Action-Sentence Compatibility Effect (ACE), Glenberg's baby.  This is where a sentence that implies motion in one direction (He pushed the box away) facilitates responses (button presses) that are directed away from the subject and interferes with responses that are toward the subject.

The ACE is a favorite of the embodied camp.  They want to argue that this means that the meaning of say push is grounded in actual pushing movements that must be reactivated to accomplish understanding.  The ACE is interesting but not surprising nor conclusive.  Just because two things are correlated (the meaning of the word push and the motor program for pushing) doesn't mean one is dependent on the other; one could exist without the other.  Again, think "fly", "slither", "coil", etc. etc.  Or think of it this way.  If I blew a puff of air in your eye every time I said the phrase "there is not a giraffe standing next to me", before long I could elicit an eye blink simply by uttering the phrase.  Furthermore, I could probably measure a There-Is-Not-A-Giraffe-Standing-Next-To-Me-Eyeblink Compatibility Effect (the TINAGSNTMECE) by asking subjects to respond either by opening their eyes wider or by closing them to indicate their decision. This does not mean that the eye blink embodies the meaning of the phrase.  It just means that there is an association between the phrase and the action.  Glenberg's ACE simply highjacks an existing association that happens to involve action-word pairs that have not only a "pragmatic" association but also an "epistemic" relation, to use their terminology, and calls them one and the same.

Another study that GandG highlight as further evidence for an ACE-like effect makes my point.   Here is the relevant paragraph:

Zwaan and Taylor (2006) obtained similar results using a radically different ACE-type of procedure. Participants in their experiments turned a dial clockwise or counterclockwise to advance through a text. If the meaning of a phrase (e.g., “he turned the volume down”) conflicted with the required hand movement, reading of that phrase was slowed.
Unlike in Glenberg's ACE procedure, Zwaan and Taylor showed that arbitrary pairings between phrases and actions show the same effect (more like the eyeblink example).  Yes, some volume controls involve knob rotation, but others involve pressing a button, increasing/decreasing air pressure passing through the larynx, covering or cupping your ears, or placing your hand over your friend's mouth.  When you read the phrase, "he turned the volume down" did you simultaneously simulate counterclockwise rotation, button pressing, relaxation of your diaphram, covering your ears, and covering your friend's mouth in order to understand the meaning of the phrase?

GandG also selectively site data in support of their claims while obscuring important details:

Bak and Hodges (2003) discuss how degeneration of the motor system associated with motor neuron disorder (amyotrophic lateral sclerosis -- ALS) affects comprehension of action verbs more than nouns.

This is true statement.  What is lacking, however, is the fact that Bak and Hodges studied a particular subtype of ALS, that subtype with a dementia component.  In fact, high-level cognitive and/or psychiatric deficits appear first in this subtype with motor neuron symptoms appearing only later.  I'll let Glenberg and Gallese tell Stephen Hawking that he doesn't understand verbs anymore.

So much for the first two sections.


Rolf Zwaan said...

Nice post but your description of our finding is silly of course. Since when does "he turned the volume down" mean "he relaxed his diaphragm" or "he covered his ears", let alone "he covered his friend's mouth"? Needless to say we were aware that turning down the volume could be associated with button pressing, which is why we performed a norming task.

Otherwise, my current view of the motor system in language comprehension may not be too far from yours. http://rolfzwaan.blogspot.nl/2013/03/assessing-armada-language-comprehension.html

Greg Hickok said...

Hi Rolf,

Let me throw some silliness back at you. If you are talking to loud and I say "turn down the volume", will you start looking for a rotary volume control? Or will you somehow know that you should relax your diaphragm?

Or if your iPhone is ringing loudly during a lecture and I say, "turn down the volume" will you say, "I can't, there's no rotary volume control on this thing!"?

More generally, tell me this: what motor representation can you point to that will capture the meaning of "turn down the volume"?

Greg Hickok said...

The point is that while we *can* associate the meaning of words and phrases with various actions, this does not mean that the action codes are in any way part of the meaning. Your experiment demonstrates this quite nicely I think because you used an arbitrary pairing between meaning (decreasing perceived sound level) and movement (rotating a dial).

If you want to say that because some human cultures have instantiated this arbitrary pairing in their devices, dial rotation *movements* are now part of the meaning for those humans who actually use such controls, then I won't argue with you. I will ask you, though, how much of the meaning of the phrase are you actually explaining with dial rotation or more precisely with the motor program that codes it? Put in statistical terms, maybe you are significantly explaining some of the semantic variance, but if all you've explained is .001% of that variance, how useful is this research? Wouldn't it be a better use of our scientific efforts to figure out what is semantically in common with turning down the volume on radios, iPhones, and voices, and doing so in the astronomically large number of motoric ways that the goal can be achieved?

Rolf Zwaan said...

You’re a little hung up on the turning the volume down example. Of course we used other examples, such as screwing in a light bulb, opening a pickle jar, and so on. For each of our items, manual rotation was viewed as the most typical action (in a pilot experiment). I realize that you can open a pickle jar by dropping it out of an airplane, blasting it with a blow torch, or having an elephant sit on it, but somehow our subjects didn’t think of these options. A sentence activates a set of motor representations with the more typical ones receiving more activation than the less typical ones.
As I say in my blog post, though, I think the role of motor representations is limited in language comprehension and that they have a special role to play http://rolfzwaan.blogspot.nl/2013/03/assessing-armada-language-comprehension.html.

I agree that perceptual representations are more important than motor representations. I also agree that goal representations, which you refer to in your comment, are much more important. Of course, I have written much about this in the past, for example in my 1998 Psych Bulletin article. In the post I refer to this work.

Rolf Zwaan said...

Regarding your point about meaning, my claim is that the motor representation is (at least for people with the action in their repertoire) part of the mental representation that is formed when people read the sentence "he opened the pickle jar." Specifically, it is constitutive of a deeper understanding of the action because it is part of a first-person mental simulation of the action.

As I point out in the blog, though, I think that discourse is usually about bigger things than simple actions.

Greg Hickok said...

What motor representation, Rolf? Some program for twisting counter-clockwise? What does that add beyond visual and somatosensory representation of the act which can also specific twisting counter-clockwise? How does the motor representation you will specify in your next post deepen understanding?