A r c h i v e d  I n f o r m a t i o n

Executive Summary

The literature on the uses of peer review sets forth several commonly cited goals of the peer review process that provide a context for the present study. They include efficacy—ensuring that scientifically appropriate research is supported; efficiency—ensuring that the research selected is cost-effective; fairness—ensuring that bias is removed from the application review process; and public accountability—ensuring that the public will is reflected and the public interest protected in the selection of grantees.

On March 31, 1994, Public Law 103_227, which includes Title IX, the Educational Research, Development, Dissemination, and Improvement Act of 1994, was signed into law. This legislation mandated that the Assistant Secretary of the Office of Educational Research and Improvement (OERI), in consultation with the newly established National Educational Research Policy and Priorities Board (NERPPB), develop standards to govern the conduct and evaluation of all research, development, and dissemination activities carried out by OERI, and to ensure that such activities meet the highest standards of professional excellence. These standards were to be developed in three phases, the first of which would address OERI's process for peer review. According to the legislation, peer review standards would, at a minimum, describe the general procedures to be used by each peer review panel in its operations; describe the procedures to be used in evaluating applications for grants, cooperative agreements, and contracts; and specify the criteria and factors to be considered in making such evaluations.

In accordance with these legislative requirements, the Assistant Secretary established final regulations to set standards for the evaluation of applications for grants and cooperative agreements and proposals for contracts, to take effect October 16, 1995. These standards were developed by the Assistant Secretary, in consultation with NERPPB. They were first published in a notice of proposed rulemaking in the Federal Register, June 7, 1995, to invite comments from interested parties. NERPPB gave final approval for these standards in fiscal year (FY) 1996 after all public comments had been received, discussed, and addressed as appropriate (Code of Federal Regulations [CFR], Title 34, Part 700). To date these standards for peer review have been applied in 2 years of competitions.

This study is a retrospective review of the implementation of OERI/NERPPB standards for evaluation and peer review of grant and cooperative agreement applications. The review focuses on 2 years of competitions—FY 1996 and FY 1997—and two types of competitions, those held for field-initiated studies (FIS) and for research and development centers. The study includes 20 randomly selected FIS panels and all center competitions. The review was carried out by OERI under contract, and was overseen by an expert panel that helped frame the study, direct its progress, and craft its recommendations. ¹ These recommendations are designed to assist OERI and NERPPB in considering whether to make changes in the standards or their application. This summary highlights the study recommendations that are most central to improving the peer review process.

First, standing panels of 25 to 30 reviewers should be established in each institute. Reviewers should be carefully selected to ensure that each meets the criteria established by the standards. Panels should be constituted to ensure ethnic, racial, geographic, and gender diversity. Moreover, a balance between senior and junior scholars should be sought to provide professional development for junior scholars and "enliven" the thinking of senior scholars. Proposed panelist slates should be approved by the institute directors and the Assistant Secretary for OERI, with consultation from the NERPPB.

The reviewers on these standing panels should serve set (e.g., staggered 3-year) terms and form the core of reviewers for each institute. For the center competitions, a subset of standing panelists should be used. Decisions about which panelists to select for a center competition and the number needed should be based on the applications received for a particular competition. The subpanelists could also serve as midterm reviewers, thus ensuring consistency in the review process.

For FIS competitions, there are two options for the review process. The first would entail the formation of six- to eight-member subpanels from the membership of the standing panel; these subpanels would provide the first tier of review. The first-tier review process would function much like the current process, except the subpanels would comprise primarily standing panelists and would be expanded from three members to six to eight members to provide a broader context for the review. Applications would be allocated to subpanels on the basis of the panelists' subject area expertise and experience. If the review of some applications required special technical expertise, the subpanels could be supplemented with ad hoc reviewers. During each review cycle, the team leaders of each subpanel would meet an additional day for a second-tier review to rate all the top-ranked applications from the first-tier subpanels.

A second option for the FIS review process resembles the process used at the National Institutes of Health (NIH), where the entire panel reads all applications. At NIH it is typical for a group to review 75 to 100 applications at each meeting. Each member is asked to prepare detailed reviews for a dozen or more applications. The meetings are conducted by a chair who is a peer, assisted by a staff member. Those preparing the written reviews lead the discussion of the applications assigned to them. Each application is discussed and considered. Decisions not to recommend for further consideration are made by majority vote.² If a member disagrees, he or she can submit a minority report, and when there are two or more dissenting members, a minority report must be drafted. Members who cannot assess the merits of a proposal can abstain from voting, although abstentions are not encouraged. Review of applications can also be deferred (perhaps for a site visit or to obtain additional information).

Those applications not rejected or deferred are assigned a priority score by each member. These scores are averaged by the staff member after the meeting. In addition, a summary statement for each application is prepared for transmittal to the council and the applicant by the staff person involved in the review. The statement includes a percentile ranking for the application against a reference base of all applications reviewed by the committee over three meetings, including applications not recommended for funding or deferred. The written comments of panel members and the panel discussions are the basis for these summary statements.

Standing panels might also be involved in other activities, such as recommending how OERI could help foster research in a particular area in which good applications had not been received; helping to select new panelists and ad hoc reviewers; and reviewing grant-produced products, especially once the Phase III standards (for review of completed research) have been put in place. Panels might also provide continuity in the assessment of applications so that rejected applications that had been revised and resubmitted would be reviewed by at least some of the same people. In addition, panelists could serve on midterm review teams for existing centers.

Given that some institutes receive up to 200 applications annually, methods for reducing reviewer workload should also be considered. Several possibilities are elaborated in this report. They include, for example, the use of preliminary reviews to reduce the number of full applications receiving a detailed evaluation, and the use of pre-applications, with only a subset of applicants being asked to prepare a full application.

This report also makes other recommendations for improving the OERI peer review process. First, professional development should be enhanced for the U.S. Department of Education (ED) staff, especially those new to the process, as well as for applicants and reviewers. Additionally, reviewers would benefit from questions to guide their reviews and from elaborated scoring rubrics. The standards would benefit from clarification in several areas: the term "multiple perspectives" should be further defined to ensure that panel membership is balanced by disciplinary background, theoretical orientation, methodological approach, and research role, as well as gender, race/ethnicity, and geographic location; conflict of interest should be defined to include professional relationships as a source of conflict; and priorities should not be listed for FIS competitions. Modifying the review criteria and weightings would also enhance the process: the meaning of "national significance" should be clarified; for center competitions, the project design criterion should be elaborated; and weighting for the management criterion should be increased. Standardization of scores should be eliminated as well; the use of second-tier panels and standing panels would make this process unnecessary. Finally, more detailed feedback should be provided to unsuccessful applicants, and the use of technology in the peer review process should be explored.

[ Acknowledgements ]

[ Overview ]