Thursday, March 10, 2011

On the relation between language and music

Much has been said on the relation between music and language. Most of it arguing for common computational foundations:

...syntax in language and music share a common set of processes (instantiated in frontal brain area) - Patel, 2003

...some aspects of structural integration in language and music appear to be shared -Fedorenko, et al., 2009

All formal differences between language and music are a consequence of differences in their fundamental building blocks (arbitrary pairings of sound and meaning in the case of language; pitch-classes and pitch-class combinations in the case of music).† In all other respects, language and music are identical. -Katz & Pesetsky, 2011

This claim predicts that there should be overlap in the brain regions involved in processing both language and music. Surprisingly, no one has assessed this directly in the same subjects. Until now.

A new paper in the Journal of Neuroscience reports a study in which participants listened to either sentences or melodic stimuli during fMRI scanning (Rogalsky et al., 2011). Overlap between the two conditions was, in fact, observed, but only in relatively early auditory areas (not surprisingly, given that both stimuli are acoustic). No overlap was found in regions thought to be involved in structural processing, i.e., Broca's area and the anterior temporal lobe. Language activated a more lateral temporal lobe network, while music activated a dorsomedial pattern in the temporal lobes. Broca's area, a prime candidate for structural processing according to many, was not reliably activated by either stimulus class, once acoustic envelope information was controlled. Even within the region of overlap in upstream auditory areas, pattern classification analysis revealed that music and language activated a different pattern of activity.

What does this mean? Despite the recent hype, I don't think structural processing music and language have all that much in common, at least in terms of neural resources. I think previous behavioral and electrophysiological evidence for shared resources has more to do the tasks employed (typically violation studies) than normal structural processing itself.

Fedorenko E, Patel A, Casasanto D, Winawer J, & Gibson E (2009). Structural integration in language and music: evidence for a shared system. Memory & cognition, 37 (1), 1-9 PMID: 19103970

Patel, A. (2003). Language, music, syntax and the brain Nature Neuroscience, 6 (7), 674-681 DOI: 10.1038/nn1082

Rogalsky, C., Rong, F., Saberi, K., & Hickok, G. (2011). Functional anatomy of language and music perception: Temporal and structural factors investigated using fMRI. Journal of Neuroscience, 31(10): 3843-3852


David Pesetsky said...

Are the melodies used in your study available somewhere? I'm puzzled by the only description of them in the article: "The composition of each melody outlined a common major or minor chord in the system of tonal Western harmony, such as C major, F major, G major, or D minor."

Was there really only one chord implicated in each melody?

Greg Hickok said...

Chord looks like a typo. Should have said "key". I'll see if I can get some sample stims posted so you can hear them.

David Pesetsky said...

That would be great, thanks.

Bob McMurray said...

Great post -- nice to see someone questioning this superficially very appealing story about language and music. You might want to check out a behavioral study I and some of my students recently did showing strikingly different patterns of results in music and speech for tasks involving the integration of context and expectations (McMurray, Dennhardt & Struck-Marcell, 2008, Cognitive Science, 32[5]).

Marcus Pearce said...

Interesting paper. The references cited (Patel, 2003; Fedorenko et al. 2009) as well as other related research (e.g., Steinbeis & Koelsch, 2008;Jentschke et al., 2008) observe processing interactions (behavioural and electrophysiological) between language (specifically syntax) and chord sequences differing in the degree to which they follow the rules of musical harmony.

This may account for the failure to find any overlap in the current study which used short (mean 3s) melodies (as opposed to chord sequences) all of which followed tonal rules.

Jentschke, S., Koelsch, S., Sallat, S., & Friederici, A. D. (2008). Children with specific language impairment also show impairment of music-syntactic processing. Journal of Cognitive Neuroscience, 20, 1940–1951.

Steinbeis, N., & Koelsch, S. (2008). Shared neural resources between music and language indicate semantic processing of musical tension-resolution patterns. Cerebral Cortex, 18, 1169–1178.

Marc Ettlinger said...

I was wondering (hoping) if this was going to be discussed here when I saw it in the JNeuro TOC. I found the No Broca's (for language) results particularly exciting and this fits nicely with previous Rogalsky&Hickok collaborations arguing that Broca's is associated with task and not grammar, per se.
I did have two comments/concerns, though.

The first is that, in some sense, you're arguing from null results. Absence of evidence for Broca's activation during grammatical and/or musical processing doesn't equal evidence that Broca's area is not part of the grammatical component of language. There could be many reasons why it (Broca's in language or music/language overlap) wasn't found. One, of course, could be threshholding. It'd be interesting to see more analyses, especially since MVPA can sometimes seem like a blackbox to me.
Was there anything interesting below corrected statistical threshholds in the IFG?
What are the results of a simple ROI analysis of Broca's comparing the 3 conditions?

Second, and relatedly, I can see how the experimental manipulation may be up for some criticism. Jaberwocky minus random words may or may not reflect grammar, especially if you take a lexicalist view of phrase structures. It's tough to really judge what's going on with only the one example stimulus sentence.

At the same time, most of the evidence for music-in-broca's does come from a violation-style paradigm (e.g., Maess B, et al (2001) Musical syntax is processed in Broca's area: an MEG study. Nat Neurosci 4:540-545.), which reflects possibly weirder assumptions about what grammar is. However, there are studies, like Friederici, A.D. et al. (2006) Processing linguistic complexity and grammaticality in the left frontal cortex. Cereb. Cortex 16, 1709–1717, that one has to contend with if you want to rule out Broca's from grammar and relegate it to task or WM effects.
I did think it was one of those "no one's done this before?" studies, which attests to its simplicity and elegance. This is definitely a finding to contemplate.

Trey Jones said...

Were the participants in the study sufficiently musically savvy to perceive structure in music? If not, did they play speech in a language they don't understand? I don't speak a lick of, say, Mandarin, so Mandarin is made up of sounds for me—it has rhythms, ebbs and flows, highs and lows—much like music. (I also have no musical training, so I don't hear much structure in music, either within a moment or over time. I do hear structure in lyrics, though.) I could at least try describe the sounds of Mandarin to someone who never heard it. I can't describe Spanish because I speak it well enough that it sounds like *language*, not *sounds*. I hear meaning, not sounds. I can't imagine what English sounds like to a non-native speaker, because I think in it much of the time. It doesn't have a sound, it is thought. Would someone conclude that English and Mandarin are not processed by the same part of the brain if the participant in the fMRI machine was monolingual in one of the two languages?

Greg Hickok said...

Wow. There seems to be some interest in this study. :-) Ok, I'll start working through the excellent comments/questions.

Marcus: It is possible that longer or more complex musical sequences could drive "syntactic" areas. But I have a question: do you think listeners don't analyze shorter melodies hierarchically?

For the record, I'm not an expert on music. For this reason, I had asked Ani Patel to look at our stims and tell me whether they were sufficient to drive the kind of hierarchical processes that everyone is talking about. He thought they would. Still working on getting stims posted...

Anonymous said...

It did not occur to you to speak with a music theorist about your stimuli? Is there no music department at your uni?

Greg Hickok said...

Hi Marc,
Mostly we are arguing from non-overlap, not null results. The only place where we saw significant overlap was in relatively upstream auditory regions. Maybe this is where syntax is being computed, but I don't think so.

As far as Broca's area goes, I think the bigger problem for anyone who wants to claim that Broca's area is involved in basic structural processing is the fact that Broca's patients can make grammaticality judgements pretty darn good.

Greg Hickok said...

As for Friederici et al. 2006, the finding was that activation in ~BA44 in a reading task increased as a function of grammatical complexity. As complexity increases, comprehension becomes more difficult. What do we do when we read a sentence and don't quite get the meaning? We reread it. If this is not an option as in the Friederici et al. study, we would replay it in our heads. So it is grammatical complexity? Or is it the fact that Ss are more likely to rehearse more difficult sentences. Rogalsky & Hickok looked at this question directly and found that the BA44 "complexity effect" went away when you control for subvocal articulation.

Greg Hickok said...

Trey, you're not giving yourself enough credit. If you enjoy music, you hear plenty of musical structure, you are just don't have explicit knowledge of what that structure is. The same holds for language. Most people have no idea that there even IS structure in human language, but it is this structure that drives language processing.

Greg Hickok said...

Anonymous: Our goal was not to develop musically complex stimuli of a particular sort, but to look at the brain response to simple melodic stimuli. Our assumption, apropos to Marcus' comment, was that like with language listeners cannot help but analyze a melodic string of tones using whatever structural processing mechanisms they possess.

Marc Ettlinger said...

Non-overlap is a null result in a sense: you didn't find overlap.
Semantics aside, are you able to share a simple ROI analysis of Broca's across the 3 conditions?

Also, I couldn't find a link to the sentences - do you have them up somewhere? Sentences like:
1) "It was the glandar in my nederop" (the one example in the text)
I would think are pretty different from
2) "A wug blarg fleegen the zith peshtel."
It is also not obvious to me how (1) and/or (2) would be cut up. Are articles and expletives considered words, etc.?

Greg Hickok said...

Marc: we didn't do an ROI analysis, but we did use a fairly liberal threshold, precisely to ensure that the non-activation was not a thresholding problem. So, I can tell you that at an *uncorrected* threshold of p = .005 there was nothing in Broca's area for sentences or music relative to *rest* (i.e., scanner noise); and this was with an N=20 subjects. At same time, there *was* highly reliable activation in the ATL (a region that tends to be relatively hard to activate) to the sentence stimuli. In other words, I don't think we are dealing with a power issue. We are dealing with the fact that Broca's area doesn't get involved with sentence processing unless the stims contain violations or things get really difficult (working memory? cognitive control?).

The sentences were more like (1) than (2). I'll dig them up too and post with the music stims.

Anonymous said...

The sentences were more like (1) than (2). I'll dig them up too and post with the music stims.

Looking forward to it.

Anonymous said...

Did you post the stimuli?

Greg Hickok said...

Sorry this took me so long. It's been a busy few weeks. Here are some sample music and speech stims:

Kathrin Rothermich said...

Very interesting study! I was wondering whether you plan to look at comparing more natural speech stimuli and not pseudosentences with music? Would you expect to find different activations/overlap if you would use lexical sentences?

Jeroen van Baar said...

Greg, I think one of your last comments here captures the issue: "We are dealing with the fact that Broca's area doesn't get involved with sentence processing unless the stims contain violations or things get really difficult (working memory? cognitive control?)." (March 14, 2011).

If we want to learn about the processing of syntax, we need to make sure the brain is engaged in syntax analysis, and I guess there are two obvious cases in which this happens; 1. when syntax is violated and 2. when syntax analysis is needed in order to extract meaning from a sentence. So if a sentence is not attended and meaning does not need to be extracted, the auditory stimulus will flow into your brain but higher-order processing will not take place, which leaves an absence of syntax-related activation. Likewise, if a sentence has no meaning at all, as in your jabberwocky stimuli, syntax plays no role in understanding the sentence anymore, so your brain will just not bother. What I'm trying to say, I guess, is that syntax serves meaning (in good Chomskian tradition), and meaningful stimuli may just engage that syntax-processing unit we are thought to have.

As for music: same story? What do you think?

Greg Hickok said...

You seem to be suggesting that syntactic analysis may not be utilized for simple, grammatical sentences. I think there is a paradox hidden in there though. You suggest that one condition in which the brain is engaged in syntactic analysis is when syntax is violated. But presumably there has to be some mechanism in the brain telling you when a violation has occurred. This implies that syntactic analysis is being carried out even when no violation occurs. So even though, "The dog chased the cat" is simple and contains no violation, the fact that we readily detect a difference between that sentence and, "The dog chase the cat" means that syntactic analysis is being carried out, no?

Jeroen van Baar said...

You're right. A syntactic monitoring system must be active at all times. But don't you think that its activity will spike when a violation is detected? In neurolinguistics this seems to be a common paradigm: in both EEG and fMRI studies, conditions of no violation are often subtracted from conditions of violation to leave the activity that is specifically concerned with the analysis (or repair) of a syntax error. I think that with such an approach, we may be more successful in identifying a musical syntactic processing system, if there is one.

Your findings, of course, suggest that if any overlap is found in activity elicited by musical and linguistic syntax violations, this overlap is not a sign of a shared syntactic integration system, but rather of a general violation-activated system of some kind. This is contradicted by a couple of studies that found interactions specifically between syntax violating conditions in music and language, and not between other violation conditions, such as timbre-related surprises and semantic oddballs (c.f. Koelsch et al. (2005), Interaction between Syntax Processing in Language and Music: An ERP Study, J. Cogn. Neurosc. 17(10): 1565-77; and Slevc et al. (2009), Making psycholinguistics musical: Self-paced reading time evidence for shared processing of linguistic and musical syntax, Psychonomic Bulletin & Review 16(2): 374-81).

All in all, do you agree that a study comparing violation vs. control within-mode (language or music) and within-subjects with fMRI could provide useful insights?

willimek said...

Music and Emotions

The most difficult problem in answering the question of how music creates emotions is likely to be the fact that assignments of musical elements and emotions can never be defined clearly. The solution of this problem is the Theory of Musical Equilibration. It says that music can't convey any emotion at all, but merely volitional processes, the music listener identifies with. Then in the process of identifying the volitional processes are colored with emotions. The same happens when we watch an exciting film and identify with the volitional processes of our favorite figures. Here, too, just the process of identification generates emotions.

An example: If you perceive a major chord, you normally identify with the will "Yes, I want to...". If you perceive a minor chord, you identify normally with the will "I don't want any more...". If you play the minor chord softly, you connect the will "I don't want any more..." with a feeling of sadness. If you play the minor chord loudly, you connect the same will with a feeling of rage. You distinguish in the same way as you would distinguish, if someone would say the words "I don't want anymore..." the first time softly and the second time loudly.
Because this detour of emotions via volitional processes was not detected, also all music psychological and neurological experiments, to answer the question of the origin of the emotions in the music, failed.

But how music can convey volitional processes? These volitional processes have something to do with the phenomena which early music theorists called "lead", "leading tone" or "striving effects". If we reverse this musical phenomena in imagination into its opposite (not the sound wants to change - but the listener identifies with a will not to change the sound) we have found the contents of will, the music listener identifies with. In practice, everything becomes a bit more complicated, so that even more sophisticated volitional processes can be represented musically.

Further information is available via the free download of the e-book "Music and Emotion - Research on the Theory of Musical Equilibration:

or on the online journal EUNOMIOS:

Enjoy reading

Bernd Willimek