Even with high scores from reviewers, the UK Medical Research Council rejected two proposals from the University of Leicester's Ian Eperon, and he set out to figure out the degree to which the agency follows reviewer recommendations in its decision-making process, Nature News reports.
After getting the data through freedom of information requests, Eperon found that for 2013 to 2014, the scores that outside reviewers gave to proposals in his field didn't seem to have an effect on whether or not they were funded.
He received data on 302 proposals in molecular and cellular medicine and saw that a number of well-reviewed proposals were rejected at the 'triage' stage, that is before being sent to panel for discussion. The proposals that were approved or rejected by that board had similar ranges of scores.
"To Eperon, this is evidence that outside reviewers' scores have little bearing on whether a proposal is ultimately successful," Nature News says. Eperon also says that the selection process should be more transparent.
The University of Nottingham's David Bates tells Nature News that Eperon's findings square with his own experiences. "Of the nine applications I've put in under the current scoring scheme, the three funded grants actually ranked in the bottom four for scores," Bates says. "My five best-scoring grants didn't get funded, but three of my four worst did."
The MRC's Nathan Richardson says that external peer review is "hugely valuable" and that the agency relies more on the written comments that reviewers provide than the score. Oftentimes, he adds, there is a "disconnect" between the score and what's written.
"[N]ot every referee makes sensible comments (any academic will attest to that), nor gives scores that match their written comments — and the job of board members is precisely to moderate this," adds the University of Manchester's Douglas Kell, the former head of the UK Biotechnology and Biological Sciences Research Council.