Wednesday, November 7, 2007

Accountability in the review process

Having been knee deep recently responding to both grant proposal and paper reviews (as reviewee), I find myself more and more annoyed with the review process. Sometimes, maybe even most of the time, reviews can be helpful. But we all get those off-base or nitpicky reviews that are at best a tedious annoyance or at worst, a grant killer. Anonymity and lack of public accountability for what one writes in a review, I think, gives some reviewers carte blanche to shoot from the hip, often causing collateral damage.

There's a solution: Make the reviews public.

Not that many people would read them. We've already got more than enough to read. But maybe just knowing that your off the cuff remarks might be subject to public ridicule -- on some blog, for example ;-) -- would be enough to induce a little constraint and rationality.

There are other benefits to public reviews. Sometimes reviewer-reviewee exchanges are highly instructive, and sometimes more interesting than what goes in the paper. It could be beneficial, or at least discussion-provoking, to see these behind the scenes debates. Published reviews could also cut down on work when responding to reviews: when you get the same criticism over and over again, you could just cite a previous review response rather than writing a whole new response every time ("see Hickok & Poeppel review response 2000, 2004, 2007 for repeated and thorough dismantling of the same tired point you raise here"). It might also promote more constructive criticism in reviews, or even more willingness to review papers because the reviewer would get some credit for suggesting that clever control or theoretical insight. Folks might even become so proud of their reviews that they might start signing them and listing them as pubs.

Maybe we'll start publishing reviews of our papers here. I wonder if that would cause a stir.


Jonas said...

Hi Greg,

Thanks for this interesting post—it spurred quite a bit of afterthought:

I like your idea of the open debate, and especially the reviews database, where we could refer our reviewers (each other, that is) to instead of, err, copying and pasting.

The only thing I would be concerned with in open reviews is not the grumpy-old-man style some of us might indulge in while feeling somewhat save in anonymity, but the serious concerns one might have with a paper:

At least in the reviews I got so far, many points raised just laid the finger in the wound, which is fair enough, and improved each and every publication one way or the other—that's what we have them for.

Speaking for myself as an occasional reviewer, I feel that less established young scientists possibly become more equal opponents in blind reviews, and their concerns might be taken as serious.
Also, do we want the reviews to become a nice meet-and-greet like politicians shaking hands for the cameras (which it could become if our names were out there immediately)? Wouldn't open reviews in our field, where everybody actually depends on these publications and grants, not lead to an increase of the (already present) strategic courtesy and leniency? And would this not inevitably lead to poorer science being published and pursued?

In sum, I wonder whether there isn't something worth keeping in this blind review procecure—otherwise scientists over the last hundred years would have gotten rid of it, don't you think? Best wishes from Talking brains Leipzig, Jonas Obleser

Greg Hickok said...

Hi Jonas,

Thanks for your comment. I like the TB Leipzig reference! We should ditch TB East and TB West terminology and just specify our city so everyone can play. That was the intension of this blog anyway -- an open forum for all Talking Brains, not just the Hickok and Poeppel show! It's getting hard to come with with stuff all on our own -- we could use some community help!

I think there are benefits to anonymity, particularly when junior researchers are involved as you pointed out, so I don't necessarily think *that* aspect of the review process should be abandoned. So reviews could be made public, but the reviewers kept anonymous, for those who wish to remain so.

Would this change reviewer attitude? I think so, at least for most people. For example, you'd probably be more careful if you knew your review would be read out loud (even anonymously) to a large crowd of colleagues -- e.g., prior to the keynote address at the CNS meeting -- than if it were only read to the author of the paper you reviewed.

Another example, suppose I started posting reviews of my papers in this blog along with my critical responses, particularly emphasizing the absurd reviewer comments. Then suppose you receive an editorial request to review a paper of mine. Wouldn't you be a little more careful to make sure your comments are solid? Of course, a reviewer annoyed at the possibility of their review being made public might work a little harder to criticize my paper, but if the critiques are valid, that's not such a bad thing.

I'm still trying to think what the consequences of pulling such a stunt (publishing reviews of my papers here) would be. Is there a journal policy against it? Would editors have a hard time getting people to review my papers? Would editors decline to deal with my submissions? Or would reviewers welcome the chance to contribute publicly to the heretofore secret "parallel literature" of peer reviews?

jonas said...

Dear Greg, I once saw the very straightforward act of a colleague, simply signing his review with his name. I was impressed by that, and maybe we should start that way: Sign it, but retain the right not to do so (cf. the junior thing above).
By the way, some journals from the Biomedcentral family actually do exactly what you suggest: They publish the reviews and the various versions of the manuscript alongside the final piece (keeping the reviewers anonymous, of course, but anyway ... see here for example—more about smoking brains, it appears). Maybe this is actually the way to go, I agree.
Best, Jonas

jonas said...

Erratum: They even publish the reviewers' names — your point. J.