Wednesday, November 28, 2012

Action Based Language - More on Glenberg and Gallese

The core of Glenberg and Gallese's proposal is that language is grounded in a hierarchical state feedback control model, made possible, of course, by mirror neurons.  I actually think they are correct to look at feedback control models as playing a role in language, given that I've previously proposed the same thing (Hickok, 2012) along with Guenther, Houde and others, albeit for speech production only, not for "grounding" anything.  Glenberg and Gallese believe, on the hand, that the feedback control model is the basis for understanding language.

Their theoretical trick is to link up action control circuits for object-oriented actions and action control circuits for articulating words related to those actions.  Motor programs for drinking are linked to motor programs for saying "drink".  Then when you hear the word "drink" you activate the motor program for saying the word and this in turn activates the motor programs for actual drinking and this allows you to understand the word.

The overlap ... between the speech articulation and action control is meant to imply that the act of articulation primes the associated motor actions and that performing the actions primes the articulation. That is, we tend to do what we say, and we tend to say (or at least covertly verbalize) what we do. Furthermore, when listening to speech, bottom-up processing activates the speech controller (Fadiga et al., 2002; Galantucci et al., 2006; Guenther et al., 2006), which in turn activates the action controller, thereby grounding the meaning of the speech signal in action.

So as I reach for and drink from my coffee cup, what words will I covertly verbalize?  Drink, consume, enjoy, hydrate, caffeinate?  Fixate, look at, gaze towards, reach, extend, open, close, grasp, grab, envelope, grip, hold, lift, elevate, bring-towards, draw-near, transport, purse (the lips), tip, tilt, turn, rotate, supinate, sip, slurp, sniff, taste, swallow, draw-away, place, put, set, release, let go?  No wonder I can't chat with someone while drinking coffee.  My motor speech system is REALLY busy!

By the way, what might the action controller for the action drink code?  It can't be a specific movement because it has to generalize across drinking from mugs, wine glasses, lidded cups, espresso cups, straws, water bottles with and without sport lids, drinking by leaning down to the container or by lifting it up, drinking from a sink faucet, drinking from a water fountain, drinking morning dew adhering to leaves, drinking rain by opening your mouth to the sky, drinking by asking someone else to pour water into your mouth.  And if you walked outside right now, opened your mouth to a cloudless sky and then swallowed, would you being drinking?  Why not?  If the meaning of drink is grounded in actions, why should it matter whether it is raining or not?

Because it's not the movements themselves that define the meaning.

But the motor system can generate predictions about the consequences of an action and that is where the meaning comes from, you might argue, as do Glenberg and Gallese:

part of the knowledge of what “drink” means consists of expected consequences of drinking

And what are those consequences? Glenberg and Gallese get it (mostly) right:

...predictions are driven by activity in the motor system (cf. Fiebach and Schubotz, 2006), however, the predictions themselves reside in activity across the brain. For example, predictions of how the body will change on the basis of action result from activity in somatosensory cortices, predictions of changes in spatial layout result from activity in visual and parietal cortices, and predictions of what will be heard result from activity in temporal areas.

So where do we stand?  Meanings are dependent on consequences and consequences "reside in activity across the brain" (i.e., sensory areas).  Therefore, the meanings of actions are not coded in the motor system.  All the motor system does according to Glenberg and Gallese (if you read between the lines) is generate predictions.  In other words, the motor system is nothing more than a way of accessing the meanings (stored elsewhere) via associations.

So just to spell it out for the readers at home.  Here is their model of language comprehension:

hear a word --> activate motor program for saying word --> activate motor program for actions related to word --> generate predicted consequences of the action in sensory systems --> understanding.

Why not just go from the word to the sensory system directly?  Is the brain not capable of forming such associations? In other words, if all the motor system is doing is providing an associative link, why can't you get there via non-motor associative links.

More to the point: if the *particular* actions don't matter, as even the mirror neuron crowd now acknowledges, and if what matters is the higher level goals or consequences, and if these goals or consequences are coded in sensory systems (which they are), then there is little role for the motor system in conceptual knowledge of actions.

Glenberg and Gallese correctly point out a strong empirical prediction of their model:

The ABL theory makes a novel and strong prediction: adapting an action controller will produce an effect on language comprehension

They cite Bak's work on ALS and some use-induced plasticity effects.  Again, let me suggest, quite unscientifically, that Stephen Hawking would have a hard time functioning if he didn't understand verbs. Further, use-induced plasticity is known to modulate response bias -- a likely source of these effects.  In short, the evidence for the strong prediction is weak at best.

But rather than adapting an action controller, let's remove it as a means to test their prediction head on.  Given their model in which perceived words activate motor programs for articulating those words, which activate motor programs for generating actions, which generate predictions etc., if you don't have the motor programs for articulating words you shouldn't be able to comprehend speech, or at least show some impairment.  Yet there is an abundance of evidence that language comprehension is not dependent on the motor system.  I reviewed much of it in my "Mirror Neuron Forum" contribution that Glenberg edited and Gallese contributed to.  NONE OF THIS WORK IS EVEN MENTIONED in Glenberg and Gallese's piece.  This is rather unscholarly in my opinion.

Toward the end of the paper they include a section on non-motor processes.  In it they write,

We have focused on motor processes for two related reasons. First, we believe that the basic function of cognition is control of action. From an evolutionary perspective, it is hard to imagine any other story. That is, systems evolve because they contribute to the ability to survive and reproduce, and those activities demand action. As Rudolfo Llinas puts it, “The nervous system is only necessary for multicellular creatures-that can orchestrate and express active movement”  Thus, although brains have impressive capacities for perception, emotion, and more, those capacities are in the service of action

I agree. But action for action sake is useless.  The reason WHY brains have impressive capacities for perception, emotion, and more is to give action purpose, meaning.  Without these non-motor systems, the action system is literally and figuratively blind and therefore completely useless.

Why the unhealthy obsession with the motor system and complete disregard for the mountain of evidence against their ideas.  Because the starting point for all the theoretical fumbling is a single assumption that has gained the status of an axiom in the minds of researchers like Glenberg and Gallese: that cognition revolves around embodiment with mirror neurons/the motor system at the core. (Glenberg's lab name even assumes his hypothesis, "Laboratory for Embodied Cognition").  Once you commit to an idea you have no choice to build a convoluted story to uphold your assumption and ignore contradictory evidence.

I don't think there is a ghost of a chance that Glenberg and Gallese will ever change their views in light of empirical fact.  Skinner, for example, was a diehard defender of behaviorism long after people like Chomsky, Miller, Broadbent and others clearly demonstrated that the approach was theoretically bankrupt.  Today the cognitive approach to explaining behavior dominates both psychology and neuroscience, including embodied approaches like Glenberg and Gallese's.  My hope is that by pointing out the inadequacies of proposals like these, the next generation of scientists, who aren't saddled with tired assumptions, will ultimately move the field forward and consider the function of mirror neurons and the motor system in a more balanced light.

Hickok, G. (2012). Computational neuroanatomy of speech production. Nature Reviews Neuroscience, 13, 135-145.


Anonymous said...

Hi Greg,

I thought about your (so numerous) reviews against mirror neurons.

I thought about your, let's say, "animosity" against mirror neurons.

Mirror neurons exist. But, you know that, thanks to you.


Greg Hickok said...

Hi Marc,
I like mirror neurons and I have no doubt that they exist, even in the humans where the direct evidence is sparse. My frustration is that mirror neuron fans systematically ignore evidence that contradicts their hypothesis. How can an individual with cerebral palsy and severe anarthria who has never spoken a word or walked, jumped, or run nevertheless understand language?

Anonymous said...

Cerebral palsy AND severe anarthira. Yes, you're probably right. But what about neural plasticity?

A French song dedicated to you:


Greg Hickok said...

So without the ability to speak or move much at all, such individuals can learn to perceive speech and understand actions using some other, non-motor route? Is that your claim? What's your evidence that this is possible? Why can't neural plasticity allow them to recover from the brain damage that destroyed their motor system?

Would you then predict that acute disruption of the motor system should prevent understanding because there is not sufficient time for neural plasticity to do its magic? Yes, you would. Been there. Did that experiment. Your prediction is falsified.

Citing "neural plasticity" is handwaving. Doesn't explain a thing.

Anonymous said...

you're problably right Greg.
but, if not.

also, "citing H&P 2007" is handwawing. Doesn't explain a thing?


Greg Hickok said...

You want to elaborate?

Anonymous said...

Not a problem.

you're an editor of many scientific journals.

you're a reviewer of many scientific journals.

you defends your model(s).

it's perfectly scientifically (un)fair.

Greg Hickok said...

One of the reasons we started this blog was to break down some of those barriers. So here and now we have an opportunity to debate the merits of the various theoretical positions in light of the data. A link to YouTube video is not a good argument. Appealing to my supposed status in the field (many would take issue) is not a good argument. I'm happy to debate but you have to give me something to work with.

Anonymous said...

I agree.

So, you're an (associate) editor of many scientific journals.

you're a reviewer of many scientific journals.

you defends your model(s).

Do you think, it's perfectly scientifically (un)fair?

Marie said...

I'm planning to write a literature review about embodied language for my MSc. Grounded cognition is something that interests and appeals to me, especially the apparent change in views from an abstract and amodal system of language representation to one which is grounded in the motor systems of the brain (seems to make sense). But there seems to be one big problem - all the 'interesting' fMRI data revealing activation in motor areas corresponding to production and perceiving language... it's just correlation. "Neurons that fire together wire together" says nothing about the functional role those connections play. My question is: how on earth could we ever untangle those correlations to suggest causation? I'm aware there is a large literature on this topic, but I was wondering if you could point me towards and papers which explicitly deal with this? Seems like the question is pretty fundamental to understanding cognition.
Best wishes,

Greg Hickok said...

The critical test is whether damage to the motor system impairs speech perception or the ability to understand actions. Here's my take on the issue:

Hickok, G. (2010). The role of mirror neurons in speech perception and action word semantics. Language and Cognitive Processes, 25, 749-776.

Murray Grossman has some data from ALS (not discussed in the above paper) that seems to point to deficits in verb (action) comprehension, but a close look reveals that M1 does not seem to be implicated. We could probably have predicted this based on the fact that Stephen Hawking seems to be able to understand verbs well enough to be a successful physicist and author.

Anonymous said...

Hi Marie,

Apart from the paper by Greg (which I sincerely think is nice but, for some reasons, does not capture the whole story), I would recommend other papers.

First, against the view that motor system might play a causal role in language understanding, see:
- Mahon, B.Z., & Caramazza, A. (2008). A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. Journal of Physiology, Paris,102(1-3), 59–70.
- Mahon, B Z., & Caramazza, A. (2005). The orchestration of sensory-motor systems: Clues from neuropsychology. Cognitive Neuropsychology, 22, 480–494.

In contrary, for behavioral findings suggesting a causal role of the motor system, see among others (the ACE effect being many times replicated):
- Glenberg, A.M. & Kaschak, M P (2002). Grounding language in action.Psychonomic Bulletin & Review, 9(3): 558-565.

Regarding pathologies affecting primarily the motor system which can lead to selective deficits in the semantic processing of pictures and individual words related to actions, you can look at:
- Bak, T.H., O'Donovan, D.G., Xuereb, J.H., Boniface, S. & Hodges, J. R. (2001). Selective impairment of verb processing associated with pathological changes in Brodmann areas 44 and 45 in the motor neurone disease-dementia-aphasia syndrome. Brain, 124(Pt 1), 103–120.
- Bak, T.H., Yancopoulou, D., Nestor, P.J., Xuereb, J.H., Spillantini, M.G., Pulvermüller, F. & Hodges, J.R. (2006). Clinical, imaging and pathological correlates of a hereditary deficit in verb and action processing. Brain, 129(Pt 2), 321–332.
- Boulenger, V., Mechtouff, L., Thobois, S., Broussolle, E., Jeannerod, M. & Nazir, T. A. (2008). Word processing in Parkinson's disease is impaired for action verbs but not for concrete nouns. Neuropsychologia, 46(2), 743–756.
- Grossman, M., Anderson, C., Khan, A., Avants, B., Elman, L. & McCluskey, L. (2008). Impaired
action knowledge in amyotrophic lateral sclerosis. Neurology, 71(18), 1396–1401.
- Ibáñez, A., Cardona, J.F., Santos, Dos, Y.V., Blenkmann, A., Aravena, P., Roca, M., Hurtado, E., et al. (2012). Motor-language coupling: Direct evidence from early Parkinson's disease
and intracranial cortical recordings. Cortex.

Finally, for evidence based on TMS or used-induced motor plasticity, you can see at:
- Glenberg, A.M., Sato, M. & Cattaneo, L. (2008). Use-induced motor plasticity affects the processing of abstract and concrete language. Current Biology, 18(7): 290-291.
- Tremblay, P., Sato, M. & Small, S.L. (2012). TMS-induced modulation of action sentence priming in the ventral premotor cortex. Neuropsychologia, 50(2): 319-326.



Marie said...

Thank you both so much for the help, the papers were very useful. Sorry I haven't been back sooner to say thanks!