Monday, October 27, 2008

Mirror neuron review reviews, to see?

Hey, Greg, are the reviews of your mirror neuron review juicy enough that they are worth posting on the blog? I don't know if it's legitimate to post reviews of a journal article on a blog. Are there guidelines about this sort of thing?

However, given what's at stake, and given how much influence the wretched mirror neuron action perception hypothesis has, it would be both intellectually helpful and sociologically fun to see such reviews and pick at them.

I'm certainly willing -- if we can agree that it's ethically defensible -- to post some of the more outrageous reviews that I've gotten. For example, that I "understand virtually nothing". Man, that hurt my feelings! Anyway, this might not be doable, although it would be a whole lot of fun.

It would be particularly interesting to find out about how your paper will be treated in subsequent rounds of pure review and the editorial process.

Maybe, in fact, the occasional readers of this blog would comment more if it meant posting one of the more bizarre reviews that they have gotten in their own research... :-) Nothing like a little levity to balance the pain of negative reviews.

6 comments:

yisroel said...

As a graduate student who hopes to move into this field professionally, I would be interested in seeing what goes on in the review process. Maybe the educational value would offset any ethical issues.

Greg Hickok said...

Yisroel: In American football, have you ever seen when the ball comes loose and a whole pile of football players pounce on the ball. The review process is kind of like what goes on underneath that pile. Lots of anonymous elbowing, low blows, cursing, insulting jokes about your mother -- that kind of thing. If you come up with ball after all that, you get your paper published.

David: Don't think I haven't thought of it! I would LOVE to publish some of my reviews. For example, I would love to publish a review of a J. Neuroscience paper that led to the paper being rejected. The review was based on a factually incorrect bit of information. I had to go through to the formal appeal process, citing and summarizing 5 papers (one that was even published in J. Neuroscience) proving that the reviewer was wrong. I won the appeal and was allowed to resubmit (basically starting all over again), but then had the privilege of paying the $100 submission fee again! So because of one person's ignorance of the literature, I suffer what turned out to be months of delay and get to pay a "fine." (The paper was more successful its second time around and is currently being revised.)

I don't think there is a formal statement from the journal publishers indicating that reviews are privileged or private information, so I don't think posting reviews would violate any policy. I'm less sure about the ethics, though. I have seen journals publish papers where the author explicitly addresses comments by a reviewer... Maybe I should just publish the review here and see what kind of backlash there is. I'll post a survey and see what TB readers think.

RE: the review of my paper... the original plan of the editor was to publish my critical review with a follow up rebuttal by a proponent of the mirror neuron theory of action understanding. This is the reason for the delay in getting reviews back -- namely, trying to find someone to agree to write a rebuttal and then actually write it. Unfortunately, it doesn't look like said rebuttal is forthcoming.

The Vlad said...

I wouldn't do this myself. People who post reviews on the web usually come off looking like bitter crackpots. You guys don't strike me as bitter crackpots! :-)

Mary Louise Kean said...

There are two reasons that I think that authors have the right to make reviews public so long as they simultaneously make their manuscripts available in the same venue.

1. Reviewers are offering their analyses and opinions to both the journal editor and the author(s) of a submitted manuscript In doing so, they give up control of circulation. The editor gets reviews for the purpose of making an assessment of the publishability of a manuscript, that is, reviews are advisory to editors but not necessarily determinant of editorial decisions. Reviews are shared by editors with authors so that they can understand the basis for the editorial decision and any recommendations for revision, re-envisioning the problem, etc. Ultimately, authors are the end users of reviews. Any editor or reviewer who does not think that authors not only frequently ask colleagues to read pre-submission versions of their papers but also for advise in addressing reviewers comments is terminally naïve. What colleagues authors choose to ask for advise on assessing reviews is totally up to the authors. Given that, I see no reason why an author might not select a broad selection of colleagues.

2. Who suffers if reviews are made public in this way? Well, obviously, an author may be mightily embarrassed if the weight of public opinion is with the reviewers. And, as Vlad mentions, the author could come off looking like a crank. Careless reviewers may be held up to ridicule or shame if they have not fulfilled their responsibilities in a professional manner and have not taken pains to cloak their anonymity if that is their concern. Since the duty of reviewers is to assess the public scientific worth of a manuscript, responsible reviewers should have no concerns if they have fulfilled that responsibility to the best of their abilities. It seems to me that the person who has the most at stake when a rejected manuscript and its reviews are made public is the journal editor because in doing so the evaluation process of a particular journal is being put on public display. As with reviewers, I can see no reason why responsible editors need be concerned about the product of the review process seeing the light of day.

The danger of publishing reviews is that some editors and/or reviewers may decide not to consider your future submissions.

I should add by way of disclosure that I do sign my journal reviews; some editors leave my name on while others delete it when forwarding reviews to authors.

Finally, to be fair & balanced, my oddest review and my oddest response to a review: Oddest review: a rejected behavioral paper on dyslexia where a reviewer focused on plaques & tangles in Alzheimer’s disease. I still don’t understand that one. Oddest response to a review: a 17 page letter to Jacques Mehler on why I should not make some revisions to my ’77 paper on Broca’s aphasia. I should have attended to more of the comment of an anonymous (Roch Lecours) reviewer’s comments as it would have been a better paper.

Mary Louise Kean
University of California, Irvine

David Poeppel said...

hmm, we are getting lots of mixed messages on this. *everyone* agrees that it would be interesting and educational to see. but there is lots of disagreement on the ethics of the situation.

and, most crucially, i agree with the vlad: are we bitter crackpots? well ... i have my moments.

in any case, it turns out that many people remain too unaware of the peer review process, so it might be useful to discuss its ins and outs.

Greg Hickok said...

Thanks for the thoughtful comment, Mary Louise. I was thinking along the same lines. I particularly agree that making a habit of publishing reviews may make it very hard to get your papers reviewed in the future.

One of the oddest reviews I've had is of the paper David and I submitted to Neuron as a "mini-review": one reviewer recommended "reject" on the basis that the claim was common knowledge and therefore a waste of journal space, and the other reviewer recommended "reject" on the basis that the claim was clearly empirically false. The editor rejected the paper saying that it is too controversial for a mini-review. This paper ended up going to TICS where it also was deemed empirically false. We argued our case, and after three rounds of reviews/revisions it was eventually published as an "opinion." The claim we were making was that speech perception is bilaterally organized. This was our 2000 paper.