What's the real lesson from a recent sting operation that revealed that more than 150 open-access publishers readily accepted a spoof manuscript for publication? Not that open access publishing is inherently flawed, according to Public Library of Science founder Mike Eisen, but that "peer review is a mess."
Science last week published the findings from the sting experiment, conducted by John Bohannon, a correspondent in the journal's news arm. The findings are remarkable: of the 304 journals to which Bohannon submitted the "fatally flawed" paper, 157 accepted it. Only 106 journals "discernibly performed any review," of which 70 percent eventually accepted the paper. Furthermore, only 36 journals provided review comments questioning the paper's scientific shortcomings, and 16 of those papers were accepted by the journals in spite of the bad reviews.
The article has faced criticism from the open access community since Bohannon did not submit the paper to any subscription-based publishers, so the study lacked a control group. Furthermore, some have accused Science of misrepresenting the results to indicate that the issues uncovered by the study are endemic to open access publishing. Indeed, the journal's press pitch for the article was titled, "Spoof Paper Reveals the 'Wild West' of Open-Access Publishing," and it noted that the results expose "the dark side of open-access publishing."
To address some of these concerns, Science yesterday hosted a live chat with Bohannon, Eisen, and David Roos, a biologist at the University of Pennsylvania who worked with Bohannon on the sting project.
In the chat, Eisen noted that the study did a good job of exposing the fact that there are "bad actors" in the publishing world but said he's concerned that the story was "spun" to indicate that the problem is specific to open access publishers.
Eisen said that the sting was a "loaded experiment" because it didn't include any subscription-based journals. "It's true that there are open access journals that are problematic, but the way a lot of people read this, it seemed like this was a problem specific to open access journals," he said. "The study in no way showed that to be true."
Rather, he said, the results indicate that "this is a systemic problem in publishing."
Both Eisen and Roos said that the issues exposed in the study are not with open access, but with the peer-review process in general.
"The problem with peer review is almost identically problematic with subscription journals as it is with open access," said Roos.
The "real lesson" from the study, according to Eisen, is that "peer review is a mess. We need to focus on how to make that more effective."