Does climate change really happen? (2024)

Professional Member*

Credit...Tony Cenicola/The New York Times

By Aaron E. Carroll
Nov. 5, 2018
Even before the recent news that a group of researchers managed to get several ridiculous fake studies published in reputable academic journals, people have been aware of problems with peer review.

Throwing out the system — which deems whether research is robust and worth being published — would do more harm than good. But it makes sense to be aware of peer review’s potential weaknesses.

Reviewers may be overworked and underprepared. Although they’re experts in the subject they are reading about, they get no specific training to do peer review, and are rarely paid for it. With 2.5 million peer-reviewed papers published annually worldwide — and more that are reviewed but never published — it can be hard to find enough people to review all the work.

There is evidence that reviewers are not always consistent. A 1982 paper describes a study in which two researchers selected 12 articles already accepted by highly regarded journals, swapped the real names and academic affiliations for false ones, and resubmitted the identical material to the same journals that had already accepted them in the previous 18 to 32 months. Only 8 percent of editors or reviewers noticed the duplication, and three papers were detected and pulled. Of the nine papers that continued through the review process, eight were turned down, with 89 percent of reviewers recommending rejection.

Peer review may be inhibiting innovation. It takes significant reviewer agreement to have a paper accepted. One potential downside is that important research bucking a trend or overturning accepted wisdom may face challenges surviving peer review. In 2015, a study published in P.N.A.S. tracked more than 1,000 manuscripts submitted to three prestigious medical journals. Of the 808 that were published at some point, the 2 percent that were most frequently cited had been rejected by the journals.

An even bigger issue is that peer review may be biased. Reviewers can usually see the names of the authors and their institutions, and multiple studies have shown that reviews preferentially accept or reject articles based on a number of demographic factors. In a study published in eLife last year, researchers created a database consisting of more than 9,000 editors, 43,000 reviewers and 126,000 authors whose work led to about 41,000 articles in 142 journals in a number of domains. They found that women made up only 26 percent of editors, 28 percent of reviewers and 37 percent of authors. Analyses showed that this was not because fewer women were available for each role.

A similar study focusing on earth and space science journals found that women made up only about a quarter of first authors and about 20 percent of reviewers. They had higher acceptance rates than men, though.

In 2012, the journal Nature undertook an internal review of its peer review process, finding balance in its editors and reporters but disparities elsewhere. In 2011, women made up only 14 percent of the more than 5,500 peer reviewers for papers. Only 18 percent of the 34 researchers profiled in 2011-12 were women, and only 19 percent of the articles written for the “Comment and World View” section were by women.

It’s possible women declined opportunities to review, but studies have documented that male editors tend to favor male reviewers. This year, Nature reported that it had increased participation of women in the “Comment and World View” section to 34 percent, while the percent of reviewers had climbed only to 16 percent.

Unesco estimates that women make up 29 percent of the worldwide science work force.

But there are also data to support the value of peer review. A 1994 study, published in Annals of Internal Medicine, reviewed the quality of papers submitted to the journal before and after the peer review and editorial system. Researchers used a tool that assessed the manuscript’s quality on 34 items, and their work showed that all but one got better. The biggest improvements were in the discussion of a study’s limitations, its generalizations, its use of confidence intervals and the tone of the conclusions. Probably none of these would have occurred without the nudge of peer review.......

https://www.nytimes.com/2018/11/05/upshot/peer-review-the-worst-way-to-judge-research-except-for-all-the-others.html