Thursday, May 5, 2011

Revamping the scientific review process

The peer review system for journal publication works reasonably well, but is far from perfect. A major problem is the delay in review time, often caused by the difficulty of finding reviewers as well as reviewers failing to get their reviews in on time. I raised this issue previously on TB in the context of possibly paying reviewers for their service in order to encourage reviewers to accept assignments. There were some interesting responses to that idea.

Now, with four overdue reviews in my Human Brain Mapping editorial cue (7, 35, 38, and 66 days late), I'm thinking more about how to encourage reviewers to get their reviews in on time, besides pestering with annoying emails.

One commenter on my previous post had a good idea: link the timing of the review process for an author's manuscript submission to how long it takes him or her to provide reviews of other authors' manuscripts. Given that multiple journals subscribe to a few edited service website/systems (e.g., Scholar One, it may be possible to track a reviewer's on-time performance and then apply that information in the review of his or her submissions. For example, if a reviewer is, on average, 14 days late, the action editor would wait 14 days before starting the review process. There would, of course, be the normal delays in finding reviewers and getting reviews back, but late reviewers would have this extra fixed delay built in.

Do you think this would motivate you to get your reviews in on time?


Dan Mirman said...

I think that's an excellent idea. The editor could consider timeliness both when sending the paper out for review and when writing the action letter. It would basically mean converting the editorial queue from a standard first-in-first-out queue to a priority queue so that the overall turn-around time for timely reviewers would be faster than slow reviewers.

Might also need to consider number of reviews completed...

Kenny Vaden said...

A reviewer-penalty pool among journals may discourage busy researchers from volunteering to review. I’d like to see rewards for fast reviews, instead of penalties that extend the process. I'd keep a review in mind for sure if there was an I-Phone drawing involved, or even a JOCN t-shirt raffle. A Cerebral Cortex t-shirt could really improve one's nerd credentials.

Greg Hickok said...

iPhone drawing maybe, but a t-shirt? Kenny you are are nerd! ;-)

Rewards for agreeing and fast reviews is a good idea, but you can't penalize people who've never been asked to review.

How about this:

If you've never been asked to review, delay = zero.

For people who've been asked to review, but have never accepted, delay = 1 week.

For people who've been asked to review and accepted at least once, your delay is calculated as follows. You start at a baseline delay = 1 week. For every ms you agree to review, you get a one day acceleration reward. Beyond this, you are rewarded or docked a day according to the sum of the number of days early or late your reviews are.

I suppose if you know you are really late all the time you are best off never reviewing and accepting the 1 week delay, but who will decline every review request?

Hauke said...

In case it wasn't mentioned yet:
An incentive solution to the peer review problem has been proposed earlier by Hauser & Fehr:
see here



Peter said...

I think punishing reviewers for not donating more work faster would probably make many reviewers bitter and lead to more careless reviews. How about restricting reviewing to more select group of reviewers who have demonstrated their ability to provide high quality reviews in a timely fashion, and then compensating them for the extra work they do on behalf of publishing companies? It maybe untenable financially, but I think it's worth at least doing the thought experiment about more drastic changes than incentives/punishments.

Thanks for bringing this up. I think it's a really important topic.


Brian Barton said...

On a related issue: what do you think about reviewers often asking, particularly in high-tier journals, for costly, time-consuming additional experiments in order to pass peer review?

An article discussing this problem:

Davide Crepaldi said...

I think I agree with Kenny, penalties might discourage people from accepting to review. I think this would also risk to make us losing good and fair reviewers (in favour of bad and unfair ones), as normally good and fair reviewers and good and fair people, and good and fair people collaborate a lot, supervise a lot, do intensive research, etc.. They're generally busy guys and might really not be able to meet strict deadlines.
One thing that I've seen to work a lot is the fear that the Editor won't wait for your review, and you'll end up having wasted a considerable amount of time. Do you think it would be feasible to capitalise on this? Something like the Editor sending a reminder to late reviewers along these lines "I've got two reviews now, I can't wait more than 3 days for yours"...obviously in a kind of nicer way...