Everyone rants about peer review, but what isn't very well known is that the process serves two purposes, says Chris Lee at Ars Technica. It both decides the value of research — something Lee says it does very well — and serves as "a gatekeeper for scientific funding" — which, Lee says, it doesn't do as well. In the first case, the great benefit to peer review is that it examines a researcher's methods well, Lee says. And although reviewers sometimes get it wrong on the results or aren't always well versed in the field of the research they're reviewing, "examining the methods is the most important function," he adds. That's why the peer review process gets tripped up when it comes to reviewing grant proposals, Lee suggests. "Reviewers are asked to evaluate proposed methods, but, given that the authors themselves don't yet know if the methodology will work as described, how objective can they be?" he asks. "Unless the authors are totally incompetent and are proposing to use a method that is known not to work in the area they wish to use it, the reviewer cannot know what will happen." And because money is limited, funding a poor proposal is "disastrous for all concerned," Lee says. So clearly, he adds, a change is needed in the way researchers are funded instead of dumping the problem into the peer review basket.
HT: Derek Lowe at In the Pipeline