Saturday, June 29, 2013

Visual speech input drives auditory cortex

We kind of already knew this.  For example, early imaging work by Calvert and Co. showed that portions of auditory cortex activate during lip reading and electromagnetic recordings by van Wassenhove et al. showed that adding visual speech information to an auditory speech signal speeds up the N1 component of the cortical response to speech sounds.  But there was still a little wiggle room in explaining these effects.  Yes, lip reading activates auditory cortex but maybe this is just auditory imagery and with group averaged activations how do you know where you are exactly?  And yes, the N1 is an early component but it likely reflects activity in a range of auditory regions, perhaps even including STS, so how early is it in terms of the cortical processing hierarchy?

So we (Okada et al.) decided to try to nail this issue down with the following fMRI experiment: we localized auditory cortex in two different ways in individual subjects, using an AM noise localizer scan and using an anatomically defined mask covering Heschl's gyrus.  We then measured the activity in the ROIs (left and right hemis) while listening to auditory speech only and while listening to auditory+visual speech. We found that the addition of the visual speech signal significantly up-regulated the activity of auditory cortex compared to auditory speech alone.

It's confirmed then: visual speech information drives auditory processing at the first stages of the cortical processing hierarchy.

Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S. C. R., McGuire, P. K., Woodruff, P. W. R., Iversen, S. D., & David, A. S. (1997). Activation of auditory cortex during silent lipreading. Science, 276, 593-596.

van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Procedings of the National Academy of Sciences, 102, 1181-1186.


Anonymous said...

Hi Greg,

Congrats for this nice study on AV speech perception.

You mention in your paper that 'Although previous studies of silent lip-reading have found activation in lower-level auditory cortical regions ...these studies relied on group average activation maps, which can lead to mislocalizations due to averaging error'. This is no exactly true, Pekkola and colleagues (your reference 22) used individual ROI analyses on Heschl's gyrus.

Second, Calvert and colleagues (1997, 2001) mentioned that a supra-addtive response is necessary to really determined audio-visual interactions in a specific brain area (i.e., AV > A+V). A point which is not clearly mentioned in the Discussion.



Greg Hickok said...

Right. My error in the post. We did note in the paper that they had localized activity to early auditory areas, those reports only looked at visual speech with no auditory speech. So you have to worry whether it is some form of auditory imagery.

Supra-additivity is a controversial definition of AV interactions. Would you like to argue based on this idea that what we observed is not visual modulation of the auditory response? Ie, what's the point of your comment?

Anonymous said...

I just did not understand why you did not (also) test visual-only speech perception and did not mention that point on your paper.