Take Peer Review and Make it Better?

NIGMS' Jeremy Berg has been posting some interesting data on the enhanced NIH peer review system, says Comrade PhysioProf. While he's a "big fan" of the new application format, scoring system, and critique format, PhysioProf can't help wondering if the purpose of enhancing peer review wasn't to make the system better, but to "placate the extramural community who was up in arms at the peer review outcomes that were being used to perform an intrinsically impossible task: identifying the top decile of applications." Since investigators whose grants are judged in the top quartile get mad when they're not judged in the top decile, perhaps the new peer review system is just in place to make these investigators feel that their concerns are valued and the system should be more "fair," PhysioProf says. "The most interesting, but impossible, analysis of the new reviewing system is not post-hoc analyses of what reviewers are doing, but rather a direct comparison of assigned grant percentiles between the old and new systems," he adds.