When the criteria keep changing, it can be difficult to be the best grant reviewer you can be. Last year, the National Institutes of Health put a new scoring system into place and tweaked the review criteria and formats; this year, the agency introduced shorter grant applications. One of the hopes for these changes is that they will help relieve reviewer burden.
The first step to being a good reviewer is to sit down with the eight or so grant applications that are sent out about a month before the study section for a preliminary review. Scientific reviewer officers from NIH's Center for Scientific Review assign an application to three or more reviewers whose expertise matches the contents of the proposal. "Everybody who receives them is committed to read them. It takes about several hours to read each one," says Toni Scarpa, the CSR director.
These preliminary reviews are now more focused than they were in the past. "It's bulleted whereby instead of including a lot of description, you go down to the important positive and negative for each of five scoring criteria," Scarpa says. Those criteria are: significance, investigator, innovation, approach, and environment.
From these new criteria, CSR is now seeing a change in the distribution of priority scores between June 2008 and 2009. "It's a straight, smooth curve whereas the other one was a little sigmoidal, so it was more complex before than now," Scarpa says. "The scoring is doing very well."
Another change that is just getting going is having shorter applications. Instead of the behemoth proposals of the past, applicants are now limited to 12 pages for their research strategy and four pages for their biographical sketches, among other page limitations — which should cut back on reviewing time. These changes aim to focus both the applicants and the reviewers on the impact and significance of the proposed work. "We spent two years [getting] advice from everybody in the scientific community and it was made because of that recommendation," Scarpa says. "The goal was to focus more on impact and significant, whether it is was worth doing rather than 'can we do it?' I think both things are important but if it is not worth doing, it is almost doesn't matter how you do it."
Once the critiques and preliminary scores are ready, they are uploaded to a secure website where reviewers can then see the feedback from other reviewers about those applications. If there are discrepancies among the reviews, Scarpa says reviewers can then take another look at the application to make sure they haven't overlooked something. He adds that a grant review is different than a paper review. "It's mostly about an idea and the future and the feasibility of that rather than everything that needs to be done," he says.
Then there's the study section itself, which typically lasts about a day and a half and may be either face-to-face or electronic. There, reviewers discuss the applications — often just the highly rated ones — to come up with final scores and critiques. "The critique should address 'this is what the committee likes and what it doesn't like' rather than saying 'do it this way or this way or this way.' There is some room to write that, [but] the goal was to minimize that," Scarpa says.
"[A good reviewer] is somebody that needs to be relatively broad in his or her thinking in … and understands the different aspects and appreciates the different aspects of science," Scarpa adds. "It should be somebody who is ready to listen and receive input from the others."