Greg, I think one of your last comments here captures the issue: "We are dealing with the fact that Broca's area doesn't get involved with sentence processing unless the stims contain violations or things get really difficult (working memory? cognitive control?)." (March 14, 2011).My response was this:
If we want to learn about the processing of syntax, we need to make sure the brain is engaged in syntax analysis, and I guess there are two obvious cases in which this happens; 1. when syntax is violated and 2. when syntax analysis is needed in order to extract meaning from a sentence. So if a sentence is not attended and meaning does not need to be extracted, the auditory stimulus will flow into your brain but higher-order processing will not take place, which leaves an absence of syntax-related activation. Likewise, if a sentence has no meaning at all, as in your jabberwocky stimuli, syntax plays no role in understanding the sentence anymore, so your brain will just not bother. What I'm trying to say, I guess, is that syntax serves meaning (in good Chomskian tradition), and meaningful stimuli may just engage that syntax-processing unit we are thought to have.
As for music: same story? What do you think?
Jeroen,You seem to be suggesting that syntactic analysis may not be utilized for simple, grammatical sentences. I think there is a paradox hidden in there though. You suggest that one condition in which the brain is engaged in syntactic analysis is when syntax is violated. But presumably there has to be some mechanism in the brain telling you when a violation has occurred. This implies that syntactic analysis is being carried out even when no violation occurs. So even though, "The dog chased the cat" is simple and contains no violation, the fact that we readily detect a difference between that sentence and, "The dog chase the cat" means that syntactic analysis is being carried out, no?Jeroen countered:
Greg, You're right. A syntactic monitoring system must be active at all times. But don't you think that its activity will spike when a violation is detected? In neurolinguistics this seems to be a common paradigm: in both EEG and fMRI studies, conditions of no violation are often subtracted from conditions of violation to leave the activity that is specifically concerned with the analysis (or repair) of a syntax error. I think that with such an approach, we may be more successful in identifying a musical syntactic processing system, if there is one.
Your findings, of course, suggest that if any overlap is found in activity elicited by musical and linguistic syntax violations, this overlap is not a sign of a shared syntactic integration system, but rather of a general violation-activated system of some kind. This is contradicted by a couple of studies that found interactions specifically between syntax violating conditions in music and language, and not between other violation conditions, such as timbre-related surprises and semantic oddballs (c.f. Koelsch et al. (2005), Interaction between Syntax Processing in Language and Music: An ERP Study, J. Cogn. Neurosc. 17(10): 1565-77; and Slevc et al. (2009), Making psycholinguistics musical: Self-paced reading time evidence for shared processing of linguistic and musical syntax, Psychonomic Bulletin & Review 16(2): 374-81).
All in all, do you agree that a study comparing violation vs. control within-mode (language or music) and within-subjects with fMRI could provide useful insights?
Here's the issue I'd like to bring up for discussion here: "don't you think that [syntactic] activity will spike when a violation is detected?"
This is the underlying assumption of all the studies that use the violation paradigm, but it is an empirical question and I'm not convinced we have a clear answer on this. Does syntactic theory predict a spike in syntactic computation when a violation occurs? Not really. In fact, you might argue that the syntactic mechanism shuts down and something else takes over!
The problem I have with violation studies is that the occurrence of a violation is confounded with other processes, some of which may not be syntactic at all. E.g., conflict resolution, various forms of working memory, the probability of your subject talking to him/herself may increase ("Was that a violation? Should I push the button?), etc. In short, I don't know what is being measured in the response to a violation so I don't how to interpret a neurophysiological response triggered by such a violation. If you want to map the neural system involved in violation processing, then fine, study violations. But this that really what we're after here? Or are we trying to understanding the circuits involved in syntactic computations as they are normally carried out in grammatical sentences? I'm interested in the latter and so I'm not interested at all in the response to violations. I think it is misleading.
This is a theoretical concern but it is backed up by empirical results. I think we all agree that listening to structured sentences involves syntactic computation. If we look at that activation pattern associated with listening to sentences contrasted with rest or with listening to scrambled sentences or spectrally rotated versions of those sentences, we do not typically find activation differences in Broca's area. However, the most robust site of activation in violation studies is Broca's area. Given these two facts, how can we say that the violation response reflects syntactic computation?
15 comments:
Greg, I’m excited to see that this discussion has made it to your blog’s front page. The violation paradigm should, as any widely used paradigm that may just have gotten a little too popular, indeed be scrutinized.
Considering robust findings in EEG studies, I think the fact that there’s a clear-cut difference between semantic violation ERPs (N400) and syntactic ones (P600) means that there is at least something inherently semantic or syntactic about them. So they must, to some extent, reflect syntactic and semantic processing. Moreover, a posterior P600 is thought to be a marker of grammatical “revision and repair”, and it doesn’t get more syntactic than that.
Carrying this over to an fMRI approach, perhaps contrasting syntactic violation conditions with other kinds of violations - semantic ones in language, or timbre-related ones in music - could reveal an overlap in function representing a general violation response and a difference in activation that represents specific syntactic or semantic functioning. Even overlapping activation patterns could be categorized with multivariate pattern analysis. It would, however, be very easy to get lost in a maze of blurry contrasts, and there is a limit to the number of meaningful operations one can perform on scans.
What I’m getting at is that there must be some truth in syntactic ERPs, but it is difficult to extract this truth. Perhaps a more efficient approach would be one that does not center around violation, but rather around a non-surprising difference in processing difficulty. It has been established, I would say, that mental operations that are more difficult require more processing resources. Contrasting easy to difficult (garden-path?) sentences could therefore reveal differential activation representing syntactic processing, just as it reveals differences in self-paced reading times and other behavioral measures.
Going back to what sparked this debate (the hypothesized sharing of syntactic processing resources by mental representations of language and music), it may then be possible to carry out such an investigation of easy vs. difficult in music-syntactic processing as well. And that could lead to a better understanding of shared resources in music and language.
You make some very good points. I would like to believe that violation studies provide useful information. But here's the thing that worries me: if the source of the violation signal (say Broca's area), doesn't activate during processing of a stimulus that does not contain a violation, isn't that a problem?
I'd recommend going even further back. I'd take issue with "I think we all agree that listening to structured sentences involves syntactic computation." It is far from a foregone conclusion that that is the case. At least, since the 60s there have been many influential theories in linguistics that proposed that the difference between the lexicon and syntax is one of continuum and not a clear divide. There is so much evidence for that view in language that it always amazes me how little attention this hypothesis gets. It's partly because even its proponents keep writing dictionaries and grammar books - simply because the lexicon/grammar divide is such a useful heuristic. But if experiments are based on the assumption that syntax and lexicon are completely separate, they are likely to miss important things.
If you think of language processing as being more similar to face recognition than serial computation, you can come up with interesting models that account for the violation results just as well (or at least I hope so).
Thanks for your comment Dominik. The theoretical work one places in the syntactic vs. the lexical system is certainly a matter of debate. So what's your theory of the violation results?
To be honest, I don't know enough about the violations results to formulate an explanatory theory. But from this discussion, it appears to me that we don't have the whole story yet.
I'd like to see some results from morphologically rich languages with relatively loose syntax. For instance, in the 1940s, one Czech linguist observed that Czechs are much more likely to perceive case form violations in the speech of foreigners than aspectual violations. Or rather they perceive case ending violations as errors but try to find felicitous interpretations for aspectual violations. I can confirm that this seems to be the case from informal observations but I'd be interested to know what happens under the processing violations research paradigm.
I'd like to see some explanation of violations of idioms such as "the proof is in the pudding" or novel expressions like "to sneeze the napkin off the table".
I think any processing theory will also need to account for the fact that most of the time people don't notice violations - syntactic, semantic or idiomatic.
But most importantly, I'm a bit worried about our ability to match brain phenomena with linguistic phenomena at this stage. I took a bash at explaining what I mean in more detail here: http://metaphorhacker.net/2011/03/the-brain-is-a-bad-metaphor-for-language
Hi Dominik,
It’s nice to meet a compatriot here!
As far as I know, any syntax theory has to be combinatorial. So, with respect to the violation paradigm, what differences can be found between particular theories? I think none.
Vilem
But different violations (semantic, syntactic) yield increase of activation at different sites, right? Therefore, it can't be a general "error detection mechanism" or "working memory" or anything else "general purpose XY mechanism".
Asks a non-practioner.
Tobias, Freiburg
Absolutely right Tobias. But still we don't know the relation between the violation responses and the computations that underlie sentence processing normally. In fact, as far as I'm aware, there isn't even a theory, just an implicit assumption that these signals are relevant.
@vkodytek Ahoj! I think there are a number of problems with: "any syntax theory has to be combinatorial". First, do we need a theory of syntax? Syntax as we know it now is kind of a strange mixture of word order and dependency operating at a number of levels. It's a mishmash of problems encountered during attempts to mathematicize and computationalize the study of language. The order words come in is important and so is the relationship between them. But just because of that it may not constitute a separate level of description.
Second, there are different ways to take combinatorial. In a formal sense, it requires certain properties (e.g. be describable by a tree theory). But any such view completely ignores large swathes of language. In an informal sense, it is obvious that you have to "combine" smaller bits to make bigger bits. But this can be accounted for by a theory of integration (I like blending theory, but am open to others) that is more cognitively plausible than complex serial operations with large lexicon lookups. Construction grammar shows that you can account for much more about language in this way. Of course, this does not always lend itself to computational treatment of language but that should always be an engineering problem, not one of theoretical linguistics.
@Greg Re: "an implicit assumption that these signals are relevant". I am increasingly worried that similar assumptions of relevance underlie much of modern psycholinguistic and neurolinguistic research. The consistency of results is certainly very suggestive but their close alignment with a theory of language that does not describe language very well is worrying. (Also, there's the "language as a fixed object fallacy" that Clark warned about.)
Dominik, thanks for your response.
“Syntax as a mishmash”: Yes, syntax, regardless whether English or Czech, has tiers. There is some literature indicating that morphosyntax and word order may be supported by different neural circuits. The same may be true for the violation of both.
What neuroscience needs from linguistics is what most linguists can agree on. CG is an interesting enterprise but a bit too holistic. This may be the reason it reduces the combinatorial nature of syntax to mere integration of ready-to-use pieces. Of course, the boundary between syntactic and conceptual structure/processes is fuzzy. On the other hand, syntax, unlike speech sounds categorization (see Greg’s blog of Oct 6, 2011, [http://www.talkingbrains.org/2011/10/monkeys-and-their-auditory-cortex.html]) , still appears to be uniquely human, in a sense the core of language. There is the poverty of stimulus.
And you need not know the meaning of all words, either - remember Hornicek’s (?) excuse: “I’m sorry I’m late. The shtift fell into the treab.” :-)
Vilem Kodytek
Greg, thanks for your reply. If I remember correctly, an early study by Stromswold et al. (1996, B&L 52, 452) from David Caplan's lab showed increased activation in left pars opercularis when subjects had to process center-embedded versus right-branching sentences. This would argue for a role of that region when processing syntactically complex sentences. That study did not involve violations, if I remember correctly.
I agree that we know too little to perfectly understand the meaning of increased activation for different types of violations. But suppose you found an increase of activation in region A in response to syntactic violations in a sentence and an increase in region B in response to a semantic violation. In this case, I would conclude that, say, region B was involved in any sort of processing of meaning. Activation could reflect access to stored (semantic) knowledge, detection of a violation, update of the mental model or all sorts of combinations. So, using violations could be a start, right?
I assume your critique also involves ERP data, then?
Cheers,
Tobias, Freiburg
There's plenty of evidence that parts of Broca's area (and beyond!) activate more to complex than simpler sentences defined in various ways (center embedding vs. right branching, object vs. subject extraction, long vs. short movement). All of these, however, confound difficulty with complexity and Rogalsky & Hickok have shown that a good chunk of that activation can be potentially be accounted for by articulatory rehearsal, which subjects are more likely to use on more difficult stimuli.
Yes, if different types of violations cause different brain responses then that is evidence for different systems being involved. But again, without a good theory of the response to violations, it is hard to draw strong conclusions. A start? Yes. Something to use as the foundation for a model of syntactic processing in the brain? No.
Greg, would it be reasonable to assume that, in sentence comprehension, the syntactic processing of complex grammatical sentences is (typically) automatic while that of ungrammatical sentences is (in part) conscious? If so, then re-analysis can include production, which could explain why Broca’s area is activated.
However, if Broca’s area is not involved in syntax during comprehension of grammatical sentences, how would you explain syntactic repetition suppression in the left BA44/BA6 in Menenti et al. (2011)?
Vilem
Menenti, L., S. M. E. Gierhan, K. Segaert and P. Hagoort (2011): Shared Language. Overlap and Segregation of the Neuronal Infrastructure for Speaking and Listening Revealed by Functional MRI. Psychological Science 22, 1173-1182. DOI: 10.1177/0956797611418347
It's possible there's an automatic/unconscious vs. controlled/conscious distinction here.
I've never said Broca's area is not involved in syntax. What I actually claim is "there is no compelling evidence
that there are sentence-specific processing regions [for comprehension] within Brocaʼs area" (Rogalsky & Hickok) -- a rather different claim. With that in mind, let's consider Menenti et al. The region that they find activated for syntax is posterior Broca's (~BA 44). This is the same region that activates very robustly during simple articulatory rehearsal and in fact, "syntactic effects" go away if in the BOLD signal if you control for articulatory rehearsal. In Menenti et al., subjects were listening to sentences and deciding weather the meaning of the sentence matched a picture. Question: is it possible that subjects sub vocally rehearsed at least a portion of the stimuli to help in task performance? If the answer is 'yes' (and that is the correct answer), then we have an articulatory confound that could explain the findings. I am very willing to be proven wrong, but I have not yet seen ANY evidence that convincingly shows that posterior Broca's activation is nothing more than articulatory rehearsal.
I agree, you didn't say that and I didn't intend to imply you did. I should have written "If we assume that..." I am sorry.
Yes, it is possible that the subjects rehearsed silently - though the sentences were rather short and (at least the active ones) very transparent.
Post a Comment