Transcription
NIH Peer Review:Grants and Cooperative Agreements
WelcomeThe mission of the National Institutes of Health(NIH) is to seek fundamental knowledge about thenature and behavior of living systems and to applythat knowledge to enhance health, lengthen life,and reduce illness and disability. NIH has alongstanding and time-tested system of peerreview to identify the most promising biomedicalresearch. This document provides an overview ofthe NIH peer review system, including descriptionsof its core values and safeguards on fairness.The NIH Peer Review ProcessThe NIH peer review process forms thecornerstone of the NIH extramural researchmission and seeks to ensure that applicationssubmitted to the NIH are evaluated by scientificexperts in a manner free from inappropriateinfluences. Currently the NIH handles over 80,000applications and engages over 25,000 reviewersper year.The NIH Center for Scientific Review (CSR) is thecentral receiving point for all applicationssubmitted to the NIH and to some other federalagencies. CSR’s Division of Receipt and Referralassigns each application a unique number; checksthe application for compliance with format andpolicy requirements; assigns the application to aScientific Review Group (SRG) for initial peerreview; and assigns the application to an NIHInstitute, Center, or Office for eventual fundingconsideration.Competing applications are submitted to the NIHelectronically and are received through theGrants.gov site, the central portal of the UnitedStates government for receipt of electronicapplications.The NIH is composed of 24 different researchcomponents that have grant-making authority andare called Institutes and Centers (ICs), each withits own specific research agenda. In addition toreceiving and referring all applications, CSRmanages the initial peer review (see below)NIH Peer Review: Grants and Cooperative Agreements
of most research and fellowship applications. TheNIH ICs manage the initial peer review of someapplications, primarily those with IC-specificfeatures such as those submitted in response toRequests for Applications (RFAs), velopment award applications.Two Levels of Peer ReviewIn order for the NIH to award research funds, anapplication must be approved by two levels of NIHpeer review. The two levels of NIH peer review helpensure that the assessment of scientific andtechnical merit is separate from, yet related to, thefunding decision.The first level of review (initial peer review) is anassessment of scientific and technical merit and isconducted by a Scientific Review Group (SRG)composed primarily of non-federal scientists whohave expertise in relevant scientific disciplines andcurrent research areas. Appointed members mayserve up to a six-year term and require approvalby the NIH Deputy Director [or in the case of theNational Cancer Institute (NCI), the NCI Director].The outcome of the initial review is provided to thefunding component (NIH IC), and to the ProjectDirector/Principal Investigator (PD/PI), in awritten document called the NIH SummaryStatement.The second level of review (Council review) isperformed by IC National Advisory Councils orBoards. Councils make recommendations onpriority areas of research, pending policy, andfunding of particular applications. They arecomposed of both scientific members and publicrepresentatives chosen for their expertise,interest, or activity in matters related to health anddisease. Appointed members usually serve a fouryear term (or usually six-year terms in NCI), andrequire approval by the Secretary, DHHS or insome cases the President of the United States.In addition, both levels of NIH peer review areconducted in accordance with the Federal AdvisoryCommittee Act (FACA). FACA requires that eachadvisory committee (SRG and Council) meeting beconducted in the presence of a Designated FederalOfficial (DFO), who ensures that the meeting isconducted in compliance with applicable laws,regulations, and policy. For SRG meetings, theDFO is commonly referred to as the ScientificReview Officer (SRO); for Council meetings, theDFO is commonly referred to as the ExecutiveSecretary.The IC Director makes final funding decisions, withconsideration of staff and Advisory Council/Boardadvice.Core Values of NIH Peer ReviewThe core values of NIH peer review are (1) expertassessment, (2) transparency, (3) impartiality, (4)fairness, (5) confidentiality, (6) security, (7)integrity, and (8) efficiency.These values drive NIH to seek the highest level ofscientific and ethical standards, and form thefoundation for the laws, regulations, and policiesthat govern the NIH peer review process.Expert Assessment:NIH policy requires that the scientific expertise inthe initial review panel be suitable for evaluatingthe potential impact of the proposed work. NIHrecruitsreviewerstoencompass broad and diverse scientific views, aswell as to assess specific aims and methodology.Evidence of a reviewer’s qualifications may befound in his or her publication record, researchfunding history, other scientific achievements,and/or recommendations from colleagues in thefield.Also, in certain cases, public representatives maybe recruited to provide perspective from thepatient or advocacy point of view, or individualswith knowledge of technology transfer oraccounting practices may serve as reviewers.Therefore, both CSR and the ICs recruit reviewersbased on their expertise and stature in the field,mature judgment, impartiality, and ability to workin a group. Close attention is given to balance in
committee membership. Appointmentsare made without discrimination onthe basis of age, race, ethnicity,gender, sexual orientation, disability,cultural, religious, or socioeconomicstatus.Transparency:Applications submitted to the NIHare evaluated for scientific andtechnical merit using established,published review criteria. Only thereview criteria published in a FundingOpportunity Announcement (FOA)may be used as the basis for theevaluation of applications submittedfor that FOA.Reviewers of research applicationsprovide an Overall Impact score toreflect their assessment of “thelikelihood for the project to exert asustained, powerful influence on the researchfield(s) involved.” Overall Impact is defined inspecific ways for different types of NIH fundingmechanisms. For fellowships, Overall Impact isdefined as “the likelihood that the fellowship willenhance the candidate’s potential for, andcommitment to, a productive independentscientific research career in a health-related field.”The review criteria for NIH funding mechanismsare stipulated broadly in regulation and furtherdefined in NIH policy. For example, the standardreview criteria used for all NIH researchapplications are Significance; Investigators;Innovation; Approach; and Environment. Forfellowship applications the standard reviewcriteria are Fellowship Applicant; Sponsor,Collaborators, and Consultants; Research TrainingPlan; Training Potential; and InstitutionalEnvironment & Commitment to Training.Additional criteria may be added for specialinitiatives but must be specified in the FOA.Reviewers are asked to assess additional reviewcriteria, as appropriate for the work proposed.These additional criteria include Protections forHuman Subjects; Inclusion of Women, Minorities,and Individuals Across the Lifespan; VertebrateAnimals; and Biohazards.Finally, reviewers are asked to comment on severaladditional considerations that do not factor intothefinalOverallImpactscore.Theseconsiderations include justification for anapplication from a foreign institution, specialconsiderations for select agent research, plans forsharing research resources, authentication of keybiological variables, the budget request andproposed project period. The comments from thereviewers are transmitted to the PD/PI andappropriate NIH staff through the SummaryStatement.The NIH also strives for transparency bypublicizing descriptions of our standing reviewpanels, the rosters of individuals who participateon review panels, and information on each fundedgrant. Finally, the guidelines sent to reviewers anddescriptions of the NIH peer review process areposted on the NIH websites.NIH Peer Review: Grants and Cooperative Agreements
Impartiality:Any circumstance that might introduce conflict ofinterest, the appearance of conflict of interest,bias, or predisposition into the review process byany participant in the process, must be managedto avoid inappropriate influence in the reviewprocess. Bases for conflict of interest in NIH peerreview include financial interests, embership, personal relationships, and otherinterests.Reviewers must sign pre-meeting and postmeeting Conflict of Interest certifications; Federalemployees serving as SRG members and Councilmembers must adhere also to the Standards ofEthics for Federal Employees.Also, each reviewer participating in either level ofpeer review must certify that he or she is not afederally-registered lobbyist. If a reviewerindicates his or her status as a federally registeredlobbyist, he or she may not participate in NIH peerreview.The NIH operates with a clear separation offunction for review staff and program staff. Thus,no member of the NIH extramural staff may serveas a reviewer on an NIH review panel, and nomember of the NIH review staff may participate inreview functions and portfolio management in thesame scientific area. Additionally, an individualmay not participate in both an application’s initialpeer review and Council review to avoid any oneindividual from having undue influence on theevaluation of an application.Finally, the NIH has policies for managing appealsof initial peer review based on documentable flawsin the review process. In certain circumstances,the appeals process results in re-review of theapplication. The four acceptable bases for anappeal of initial peer review are:
Evidence of bias on the part of one or morepeer reviewers.Conflict of interest on the part of one or morepeer reviewers.Lack of appropriate expertise within the SRGFactual error(s) made by one or more reviewersthat could have altered the outcome of reviewsubstantially.Fairness:All applications received for NIH review areevaluated using equivalent review processes. Forexample, the NIH uses a nine-point scoring scale(1 highest impact, 9 lowest impact) inreviewing all types of applications (with theexception of a few special initiatives.) The finalimpact score is calculated as the average ofindividual reviewers’ scores, multiplied by 10(range of 10 through 90). Numerical scores on thesame nine-point scale are assigned also to each of(at least) five “scored” review criteria. For certainfunding mechanisms, the final impact scores arepercentiled across different review panels tobalance variable scoring behaviors among SRGs. Apercentile is the approximate percentage ofapplications that received better impact scoresthan that particular application from the SRGduring the past year.Similarly, standard review criteria are used for theevaluation of all applications of a particularfunding mechanism, except for a few specialinitiatives. Because applications evaluated indifferent study sections often are considered in thesame Council meeting, the use of standard criteriafor each funding mechanism helps to ensureequitable evaluation.A written outcome of review – the NIH SummaryStatement – is provided to the Advisory Council,the PD/PI, appropriate NIH staff, and reviewers ofsubsequent resubmission applications. TheSummary Statement contains, at a minimum,written critiques and criterion scores from at leastthree reviewers, the final score or non-numericaloutcome designation, and the meeting roster. Itmay also contain a summary of the discussion atthe SRG meeting.NIH Peer Review: Grants and Cooperative Agreements
Confidentiality:In order to protect confidential information,portions of NIH review meetings (initial peer reviewand Council) are closed or partially closed to thepublic if grant applications (and contractproposals) are being reviewed or discussed. OnlyFederal employees with a need to know, reviewers,and support contractors are allowed to attend NIHreview meetings.In addition, all discussions, application materials(except those in the public domain such aspublications), and information about conflicts ofinterest and assignments of individual reviewers toparticular applications are strictly confidential. Infact, reviewers must sign a confidentialitycertification indicating that they have read andunderstand the confidentiality rules for NIH peerreview, and do so under penalty of perjury, beforeaccess to applications is granted.Council Book (ECB), another secure, onlineinformation system. The ECB is used by AdvisoryCouncil members to create queries, view basicapplication data and Summary Statements and,when appropriate, vote on applications as part ofan Early Concurrence process. The ECB also isused by NIH staff for managing council businessprocesses including special assignments, conflictsof interest, Early Concurrence, IC-specific webpages and documents.The NIH may take steps in response to a violationof the confidentiality agreement, in order topreserve the integrity of the NIH review process.Depending on the specific circumstances, suchsteps may include but not be limited to:Reviewers must keep review materials in securelocations at all times and should any such materialbe lost or stolen immediately so inform NIHauthorities. Furthermore, reviewers are expectedto report any breaches of security of which theyare aware (e.g., reviewers sharing passwords ordisclosing records outside the secured systems) tothe Scientific Review Officer or Executive Secretaryrunning the review meeting. Alternatively,reviewers may contact the NIH Review PolicyOfficer to report potential security breaches.Information that is related to potential breaches ofreview integrity and has sufficient detail to allowconsideration may be forwarded to OMA forconsideration and appropriate action. Integrity: Notifying or requesting information from areviewer’s institution.Terminating a reviewer’s service.Notifying the NIH Office of ManagementAssessment (OMA) with possible referral to theU.S. Department of Health and HumanServices Office of Inspector General (OIG).Pursuing a referral for government-widesuspension or debarment.Security:Review communications and grant applicationsare handled so as to protect sensitive data andconfidential information. Initial peer reviewmeetings use the Internet Assisted Review (IAR)system for communicating application andmeeting materials to reviewers. IAR operates witha secure internet connection that requires bothpassword protection and authorization by the SROfor reviewer access.Communications and materials required forCouncil meetings are managed in the ElectronicThe NIH is fully committed to maintaining publictrust in the NIH research enterprise by supportingour grantees in adhering to the highest standardsof research integrity. All NIH extramural staff,including review staff, participate in annualtraining in the proper handling of allegations ofresearch misconduct. Also, reviewers and Councilmembers are instructed to report any suspicion orallegation of research misconduct directly to theDFO in charge of the meeting, and to do so instrictest confidence.Each NIH IC designates a senior official, aResearch Integrity Officer (RIO), to handleincoming allegations of research misconduct andinformation concerning potential breaches ofreview confidentiality or security. Each DFO isinstructed to contact the appropriate RIO rightaway should an allegation of research misconductbe received. The DFO may decide to defer theapplication from review until the properauthorities can deliberate on the situation.
The Public Health Services Policies on ResearchMisconduct delegate authority for managinginvestigations of research misconduct to the Officeof Research Integrity (ORI) in DHHS. Allegationsthat involve NIH funding and have sufficient detailto allow consideration are forwarded by the NIH toORI for consideration and appropriate action.Efficiency:With the steadily increasing pace of biomedicalresearch, the NIH peer review system continuouslystrives to reduce the time between submission ofapplications to awards for the most meritoriousprojects. (In some cases, an accelerated schedulefrom application submission to award is mandatedin statute.) As a funding agency in the UnitedStates government, the NIH has converted a largeportion of its operations for receiving applicationsto use of the Grants.gov portal. Grants.gov is acentral receiving point for applications submittedelectronically to the U.S. federal government. Thissystem is environmentally-friendly as it avoidsreceipt of paper applications and facilitatesmovement of applications to appropriate NIH staffand reviewers. For instance, previously paperapplications were mailed or shipped to reviewersvia U.S. mail or courier, but now reviewers aregiven access to electronic copies of applicationsvia the secure IAR site. Other web-basedcapabilities allow potential reviewers to identifyconflicts of interest with particular applicationsearly in the process, so that another eligiblereviewer can be assigned to the application assoon as possible. This system also allowsreviewers to submit their written critiques directlyto the NIH for electronic production of SummaryStatements, an often-time-consuming step in thegrants process.The NIH also utilizes an online system – the eRACommons – to communicate with applicantorganizations and PDs/PIs. This system allowsinvestigators to see their Overall Impact scoreswithin three days after conclusion of the initialNIH Peer Review: Grants and Cooperative Agreements
peer review meeting, and to access their SummaryStatements when they are released.Finally, NIH has expedited production of SummaryStatements. Reviewers are provided templates fortheir written critiques and are instructed toprovide their written critiques in brief, bulletedformat rather than lengthy prose. This simpleformat change significantlyreducesburdenonreviewers and review staff.that their success and the advancement of theirscientific field depend on the rigor and fairness ofNIH reviews. The core values of NIH peer reviewthus have been internalized, and a culture offairness and honesty exists within the peer reviewcommunity. Reviewers devote weeks of their timeto participate in the appropriate and fair review ofapplications.ContinuingCompetitionMost NIH grants are issuedfor 3, 4, or 5-year periods,with progress reports andstaff approval requiredeach year before the nextyear’sfundscanbeawarded. However, at theend of that award period,an investigator who wishesto continue their researchmust submit an applicationforadditionalfunds(termed renewals) and recompete through the NIHpeer review system. At thisstage, the evaluation ofscientific and technicalmerit also includes an assessment of progressmade during the previous award period. In thisway, the NIH peer review system ensuresaccountability and support of the mostmeritorious research.The Culture of NIH Peer ReviewWhile laws, regulations, and policy are essential tothe success of the NIH peer review system, theculture of NIH peer review is almost as important.Reviewers and scientists in the community knowParticipation of Global ScientistsScientists from around the globe are recruited aspeer reviewers. Foreign scientists are recruited fortheir scientific and technical expertise and are notrecruited on the basis of their official position orduties. To accommodate the participation ofglobal scientists in peer review, the NIH hasadopted emerging technologies, reducing the needfor international travel. These technologies includethe use of Internet Assisted Meetings for onlinediscussions and video conferencing.
ConclusionBy promoting a fair and equitable competitiveprocess and by enlisting active researchers tomake the assessments of scientific and technicalmerit, NIH peer review well serves theadvancement of scientific knowledge and thehealth of the people of the United States andaround the world. Nonetheless, we always strive todo better and continually look for ways to evolvethe NIH peer review process and identify thehighest quality research as new opportunities inbiomedical science arise. Suggestions may be sentto ReviewPolicyOfficer@mail.nih.gov.You can download a copy of this document v2.pdfNIH Peer Review: Grants and Cooperative Agreements
NIH Institutes/Centers NINDSNINRNLMFogarty International CenterNational Center for Advancing Translational SciencesNational Center for Complementary and Alternative MedicineNational Cancer InstituteNational Eye InstituteNIH Heart, Lung, and Blood InstituteNational Human Genome Research InstituteNational Institute on AgingNational Institute on Alcohol Abuse and AlcoholismNational Institute of Allergy and Infectious DiseasesNational Institute of Arthritis and Musculoskeletal and Skin DiseasesNational Institute of Biomedical Imaging and BioengineeringEunice Kennedy Shriver National Institute of Child Health and Human DevelopmentNational Institute on Deafness and Other Communication DisordersNational Institute of Dental and Craniofacial ResearchNational Institute of Diabetes and Digestive and Kidney DiseasesNational Institute on Drug AbuseNational Institute of Environmental Health SciencesNational Institute of General Medical SciencesNational Institute of Mental HealthNational Institute on Minority Health and Health DisparitiesNational Institute of Neurological Disorders and StrokeNational Institute of Nursing ResearchNational Library of MedicineTop 5 NIH Web Sites to Get You StartedNIH Home Pagewww.nih.govOffice of Extramural Research Home Pagewww.grants.nih.govCenter for Scientific Reviewwww.csr.nih.govResearch Portfolio Online Reporting Tools(RePORT)www.RePORT.nih.govPeer Review Processwww.grants.nih.gov/grants/peer review process.htmNIH Peer Review Revealedwww.csr.nih.gov/video/video.asp
U.S. Department of Health and Human ServicesNational Institutes of Health2019 Office of Extramural Researchwww.hhs.gocwww.nih.gov
disease. Appointed members usually serve a four-year term (or usually six-year terms in NCI), and require approval by the Secretary, DHHS or in some cases the President of the United States. In addition, both levels of NIH peer review are conducted in accordance with the Federal Advisory Committee Act (FACA). FACA requires that each
