A Peer Review of Peer Review?

A new study in PLoS One statistically analyzes the problem of "widely varying reviews from different referees for the same exact paper," says In the Pipeline's Derek Lowe. The study is a meta-analysis of 52 papers and confirms that "yes, inter-reviewer reliability is low," he adds. Too high a level of agreement between reviewers could also be a problem — some of the reviewers may be redundant, or the pool of reviewers wasn't chosen with care, Lowe adds. So what level of agreement, or disagreement, is best? Well, good luck figuring that one out, Lowe says. Even with a close reading of all reviewer comments on a particular submission, it won't be easy to do.