Wednesday, May 9, 2012

Why are speech/language neuroscientists NOT concerned about bias in their behavioral measurements?

Perceptual decisions involve two processes, perceptual analysis and response selection, both of which can affect typical behavioral measures such as percent correct or reaction time.  Crucially, response selection is strongly affected by response bias, the criterion that a given subject sets for responding one way or another under a particular set of task conditions.  For example, if I ask you to tell me when you hear the syllable /ba/ in a noisy environment, and I tell you that I will give you $100 each time you say "yes" correctly (a hit), but will charge you $500 each time you say "yes" incorrectly (a false positive), you will be biased to be very conservative in saying "yes". I can push around your bias depending on how I set up the task.  Presumably, brain damage or temporary modulation of neural activity (e.g., TMS) could also, in principle, change the task conditions for the subject and bias responses, so we have to be equally careful when we manipulate the system neurophysiologically.

Happily, there is a well-worked out method for measuring the perceptual and decision components of the task.  It's called signal detection theory.  Yet most speech/language neuroscientist fail to use SDT methods to control for response bias.  The results of such studies are potentially contaminated.

For example, this is true of ALL of the TMS studies of the role of the motor system in speech perception/recognition.  And it is true of the latest study in this line of experiments.  I've pasted the abstract in below.

This study stimulated with theta-burst TMS the hand motor area and showed a change in response times in lexical decision to hand action verbs but not non-action verbs.  They conclude that "premotor cortex has a functional role in action-language understanding."

Here's why you can't conclude this from the study: the dependent measure, RT, is susceptible to response bias and therefore can be modulated either by perceptual/recognition processes (the process under investigation) OR by biasing the decision process.  The result is therefore ambiguous and as a result cannot lead to the conclusion the authors wish to make.

Psychol Sci. 2011 Jul;22(7):849-54. Epub 2011 Jun 24.

A functional role for the motor system in language understanding: evidence from theta-burst transcranial magnetic stimulation.


1Helen Wills Neuroscience Institute, University of California, Berkeley, USA.


Does language comprehension depend, in part, on neural systems for action? In previous studies, motor areas of the brain were activated when people read or listened to action verbs, but it remains unclear whether such activation is functionally relevant for comprehension. In the experiments reported here, we used off-line theta-burst transcranial magnetic stimulation to investigate whether a causal relationship exists between activity in premotor cortex and action-language understanding. Right-handed participants completed a lexical decision task, in which they read verbs describing manual actions typically performed with the dominant hand (e.g., "to throw," "to write") and verbs describing nonmanual actions (e.g., "to earn," "to wander"). Responses to manual-action verbs (but not to nonmanual-action verbs) were faster after stimulation of the hand area in left premotor cortex than after stimulation of the hand area in right premotor cortex. These results suggest that premotor cortex has a functional role in action-language understanding.


Anonymous said...

Yes, the conclusion by Willems et al. is somewhat surprising given that, essentially at the same time, when referring to Willems et al., Willems & Casasanto (2011) underscored that “it is essentially unclear why theta burst TMS led to faster reaction times” (p. 6). Moreover, they referred solely to a 2008 paper by Glenberg et al. for the evidence of motor system’s functional role (p. 9). Otherwise, Willems & Casasanto’s conclusions sound reasonable as a model/theory/hypothesis, a research path worth to follow.

If I may ask a question: Is the localization of McGurk effect in STS by Beauchamp et al. 2010 OK? I’d feel more convinced had they used IFG as the reference region.


Beauchamp MS, AR Nath & S Pasalar (2010): fMRI-Guided Transcranial Magnetic Stimulation Reveals That the Superior Temporal Sulcus Is a Cortical Locus of the McGurk Effect. J. Neurosci. 17, 2414–2417.

Willems RM & D Casasanto (June 3, 2011): Flexibility in embodied language understanding. Frontiers in Psychology. doi: 10.3389/fpsyg.2011.00116

Greg Hickok said...

I'm convinced that AV speech integration relies on the STS not IFG. Yes, IFG activates reasonably well to visual speech presented without the auditory signal, but it responds significantly less when that visual signal is paired with an auditory source. STS shows more activity with AV speech than A or V alone. STS is behaving like a region involved in AV integration whereas IFG is not. I agree though, it would have been nice to see what TMS to IFG does to the McGurk effect. We've done this behaviorally: rather than using an electromagnetic current, we simply asked subjects to generate speech while viewing/hearing McGurk stimuli. The idea is that concurrent speech will suppress the contribution of motor speech processes to the McGurk effect (articulatory suppression). We found no effect on McGurk fusion.

Anonymous said...

Thanks. It sounds convincing to me.

Goulven said...

Isn't the plausible response bias in Willem's study supposed to be the same between the condition of interest (action words) and the control condition ? In which case there is no reason to think it actually biased their study ?

Greg Hickok said...

It is assumed that response bias is not affected by TMS or the condition. But can we assume this? Suppose that with hand motor cortex stimulation the subject's hand is tingling a little bit. Could this bias the subject to respond differently to hand-related items? We go to great lengths in purely behavioral experiments to avoid experimenter induced biases in our subjects. (Remember cleaver Hans?!) For example, suppose I reran the Willem's et al. study but instead of using TMS to hand motor cortex, I simply placed a card on the table with the word "HAND" printed on it. Then suppose I found that responses to hand-related items were faster than non-hand related items. Is that a valid experiment? Or did I potentially bias my subjects? It's potentially biased of course! Why do we think that brain stimulation can't introduce the same kind of bias?

Unknown said...

It's really nice to a post about bias and RT - it's surprising how little this huge issue comes up in psycholinguistics / the psychology of language - nevermind cognitive neuroscience. All the time people are concluding some process or condition is "faster" than another based on RT (forget the difference between speed and a response latency) without any mention of response bias or speed-accuracy tradeoff. I wish more people were interested in this basic fallacy!