Wednesday, January 19, 2011

Should peer reviewers be paid for their work?

We'd all like our papers reviewed in a timely fashion by competent, thoughtful, and constructive reviewers. Needless to say, this doesn't always happen and it is easy to blame editors or journal policies for sluggish or unenlightened feedback on our journal submissions. While I don't like the policies of some journals/editors -- e.g., the policy of some journals to reject if any of the reviews are even just a nudge less than positive -- I can tell you as an editor, that the problem is most often the fault of the reviewers, i.e., you and me, the very people who complain about the peer review process.

Reviewing a paper is tedious, time-consuming, and generally thankless. Many of us get several review requests a month (or a week!). If you tend to review quickly and/or competently, you are more likely to get targeted for further review requests. So what do we do? We decline to review. As an author submitting a paper for review, we often think that the paper is "in review" basically from the moment of its submission. Sometimes it is possible to find reviewers within a day or two. Sometimes though, it can take weeks to find reviewers, especially if prospective reviewers don't respond to the original request for several days, or at all. For some papers I've edited, I've had to contact up to a dozen people before finally getting two reviewers to agree. Once in a while, I'd get a review back from one reviewer before I even find a second reviewer to agree to review the paper!

Then there is the delay in actually getting the reviews back. Most people are punctual, but certainly not all. If it takes a few iterations of review invitations to find reviewers and then one of the reviews is delayed, this can add up to substantial delays in response to the author even for journals that strive for rapid turn arounds.

Who ends up reviewing your paper? Well, in many cases it is junior scientists. Senior folks in the field tend to be more crunched for time and receive far more review requests. The people who end up agreeing to review (often via referrals from the first choice scientists) are more junior who have more time and more to gain (you get to add it to your CV!). Many junior scientists provide excellent reviews, but lack of experience and the broader perspective that comes with it can lead to weak reviews in various ways. This is the source of at least some of those slightly misinformed reviews we've all received.

So what can we do? For starters, I think it helps to realize that *we* are part of the problem. Next time you get frustrated with journal turnaround time, think twice about declining that next review invitation. A simple thing we all can do is respond quickly to review requests, even if it is to decline, and provide thoughtful alternative suggestions for other reviewers.

Some journals have instituted a reward system for reviewers, for example, if your review is on time you get entered into a lottery to win a prize. I don't find this particularly motivating though. Frontiers journals have an interesting policy: for every paper that comes in, every review editor (the editorial board, which is generally large for these journals) gets an invitation to review the paper. This seems to work because every time I click my response to these review requests, I get a message saying that the required number of reviewers has been found. Maybe this approach would work more generally. It is annoying to get this constant flood of invitations though, and pretty soon you realize it is easier to say 'no thanks' when you know there is a large pool of reviewers who received the request.

So what about paying reviewers? Would you be more willing to review a paper if you were paid $50 or $100 (if it was submitted on time of course!)? NIH pays grant reviewers to help offset the time commitment and effort. Why shouldn't journals pay too? Maybe they can't afford to, but they could impose a submission fee that would cover the cost of reviewer payment. Would you be willing to pay to have your paper reviewed?

What are your thoughts?

13 comments:

Anonymous said...

What you say is right.... But I do not agree.
You must consider the financial resources are not identical everywhere.
For example, here in Italy, where I live and work, the research funds have been progressively reduced over time. And what little that is given is devoured" by few and "greedy" groups.
Currently I do not get any research fund.
However, some small neuropsychological research I can still run.
If what you propose really had to come true, there are two possibilities: either I take the money to pay the reviewers from my salary (here wages are much lower than in USA) or I do not publish more.

Ivan Vankov said...

I think the submission fee should be optional - you should pay it only if you want your manuscript to be processed faster. On the other hand, I think paying reviewers won't help much because, as you mention, the problem with reviewing is not money, but time.

GamesWithWords said...

Half in jest, a friend proposed keeping track of how long it takes a given person to submit a review (e.g., 12 weeks). Then, next time they submit a paper, wait the same amount of time (e.g., 12 weeks) before sending that paper out for review.

Probbaly this is overly punitive and won't work for people who themselves don't publish much or don't publish in that journal, but I think the basic idea is similar to yours: finding a way to incentivize reviewing and reviewing quickly.

Anonymous said...

I don't think cash is the right incentive, but I think it would certainly help if there were some incentive. For example, it might be nice if I could put on my CV not just 'reviewed for journals X,Y and Z' but 'reviewed 20 papers within 2 weeks in 2009' or 'editors commendation for fast useful reviews' or something. This would probably have to be a general award across a range of journals (the Neuroscience Peer Review Consortium?) rather than specific to individual journals.

Anonymous said...

@Anonymous: if you want to add to your CV that you reviewed that many papers, it will work _against_ you instead of for you when you send that CV with a job application.

The publishers use people in academics to review manuscripts. Then they sell the product (journal subscriptions) back to academics at large profits. So, yes it is quite obvious that -- completely separate from the question whether or not money would be an incentive to review more and faster -- the reviewers should get paid for the job they do for the publisher.

Laura said...

I think it would be fair to pay reviewers for their efforts, particularly when they do so for regular (i.e., not open access) journals: these journals make money from getting authors to pay for being published (excuse me, for "printing color figures"), reviewers to do the quality control for free, and then libraries to pay again for the same authors/reviewers reading the articles. So why not pay the people who do the work?

@GamesWithWords: NYT's Freakonomics blog proposed something very similar to what you're saying - and I think it might work! http://freakonomics.blogs.nytimes.com/2011/01/12/incentivizing-peer-reviewers/

Greg Hickok said...

Lot's of good feedback here! I like the idea of there being some sort of a credit system whereby one's own review work is tied to the efficiency of the review process of the papers one submits. E.g., maybe you need x review credits even to get a paper reviewed. You earn review credits by performing reviews and you get extra credit for on-time reviews. But this would require coordinating across the explosion of journals we end up reviewing for and submitting to. And then we'd have to deal with the fact that some people get asked to review way more often than others, etc., etc.

G. Krishnan said...

I have a different opinion. As it is already commented, the option should be given to the authors. That is, if the manuscript needs to be reviewed within the stipulated time, they need to pay the review cost. Otherwise, it might take a little longer than the prescribed time period in the journal website/instructions.

My different opinion is that, there are a lot of researchers, especially from the under-developed and developing countries who put their best efforts to publish in reputed international journals. I feel, if the authors have not received any financial help for the study they are opting to publish, and if they are ready to pay for their reviewers, could the journal pay a small portion of the profit gained from their manuscript to such authors? I am sure that this will reinforce both the reviewers as well as the needy authors.

Anonymous said...

I think I strongly agree with this financial incentive proposal.
We should have it as an option - for the authors to provide
payment to expedite the review process - this will attract
quality, professional and speedy review. Professional work
warrants professional rewards.

I have been on both sides of the game. As an author, me or my lab
don't mind at all to pay for speedy & quality review.
(Journal owners hear this - another business idea!)
In a fast-paced science/engineering field I'm in - timing is often
critical.

On the other hand, as a reviewer it typically takes me 2 hrs or more
for a thoughtful review. I really want to do more, but really there is no incentive to do so consistently - especially being scientist and having a family, you are really hard pressed for time.

What annoys me, often times I need to get some paper from some journals, they charge quite expensive like $30 per pop ! Talk about the fruit of my free labor !

I wonder why there has not been any rebellion among scientific community .

subha_meter said...

I feel unlike other work, the review system should have two different credit systems for the reviewers - paying for their work done in time and awarding (certificate/memento) the reviewers for their contributions. The scientific journals anyway make a lot of money by exorbitantly charging the institution/ university libraries. I believe they can surely handle the small review cost without charging the authors.

Jesus Murillo said...

This is a subject that definitely raises contradicting and passionate opinions among scientists. The bare truth is that publishers make lots of money (really lots) selling their journals to the people that is doing most of the job: authors (that pay to get published), editors and reviewers (that do not get paid for their expert and dedicated work). Being as it is, it is still amazing to me that editors and reviewers (and I act as both) are not paid in however way for doing their duties and ensuring they are done well and on time.

Emma Bullough said...

do you think paying for peer review would allow more people more time to review papers?

ahmet okyay said...

Here is my suggestion:

When you submit your paper, you also pay $220 to the journal. If you get published, you get refund as royalty. Every paper gets 2 reviewers. Reviews are paid $100 each. Editor gets $20.