A r c h i v e d  I n f o r m a t i o n

I. Overview

On March 31, 1994, Public Law 103_227, which includes Title IX, the Educational Research, Development, Dissemination, and Improvement Act of 1994, was signed into law. This legislation mandated that the Assistant Secretary of the Office of Educational Research and Improvement (OERI), in consultation with the newly established National Educational Research Policy and Priorities Board (NERPPB), develop standards to govern the conduct and evaluation of all research, development, and dissemination activities carried out by OERI, and to ensure that such activities meet the highest standards of professional excellence. These standards were to be developed in three phases, the first of which would address OERI's process for peer review. The legislation stipulated that peer review standards would describe the general procedures to be used by each peer review panel in its operations; describe the procedures to be used in evaluating applications for grants, cooperative agreements, and contracts; and specify the criteria and factors to be considered in making such evaluations.

In accordance with these legislative requirements, the Assistant Secretary established final regulations to set standards for the evaluation of applications for grants and cooperative agreements and proposals for contracts, to take effect October 16, 1995. These standards were developed by the Assistant Secretary, in consultation with NERPPB. They were first published in a notice of proposed rulemaking in the Federal Register, June 7, 1995, to invite comments from interested parties. NERPPB gave final approval for these standards in fiscal year (FY) 1996 after all public comments had been received, discussed, and addressed as appropriate (Code of Federal Regulations [CFR], Title 34, Part 700). To date these standards for peer review have been applied in 2 years of competitions.

NERPPB has asked for a review of the operation of the standards to date. This review is being carried out by OERI under contract, and is being overseen by an expert panel that helped frame the study and directed its progress. The panel members discussed and critiqued multiple versions of the present report and approved its findings and recommendations. The panel includes the following members: Christopher T. Cross, president, Council for Basic Education; Carl F. Kaestle, professor of education, history, and public policy, Brown University; Sharon Lewis, director of research, Council of Great City Schools; Penelope L. Peterson, dean, School of Education and Social Policy, Northwestern University; and Judith Sunley, assistant to the director for social policy and planning, National Science Foundation.

Study Design

This study is a retrospective review of the implementation of the OERI/NERPPB standards for evaluation and peer review of grant and cooperative agreement applications. Specifically, the study charge is to examine whether the standards are appropriate and useful; whether they contribute to fair and high-quality competitions; and how the competitions conducted under the standards have operated and how they may be improved. As appropriate, the study charge includes making recommendations on how to configure and maintain peer review panels.

The review focuses on 2 years of competitions—fiscal years 1996 and 1997—and two types of competitions—for field-initiated studies (FIS) and for research centers. The unit of analysis is the FIS panel (generally composed of three reviewers and the applications they reviewed) or center competition (composed of five to nine reviewers and the applications they reviewed), rather than applicants or reviewers. The study includes 20 randomly selected FIS panels and all center competitions. The specific research questions, data collection methods, sampling plan, and analysis methods employed are described below.

Research Questions

The specific research questions used for the study are as follows:

•	According to the standards (and other regulations governing OERI peer review), what are the appropriate uses of peer reviewers for FIS and center competitions?
•	Is the selection process for peer reviewers comprehensive and "unbiased" (e.g., draws reviewers from a wide range of sources and perspectives, selects researchers in various stages of their careers), and does it avoid selection of individuals with conflicts of interest?
•	Are peer reviewers appropriate to the applications they review? (The standards call for appropriateness with regard to training and experience in the subject area under review, in-depth knowledge of policy and practice in education, and theoretical or methodological approaches in the subject area under review.)
•	Are peer reviewers adequately instructed or otherwise "trained" for the review process, and is the review process for each type of competition (e.g., mail, in person, one/two stage) carried out effectively?
•	Does the peer review process yield reviews that provide the information needed to make funding decisions? That is, do the reviews supply the necessary content, including concise written comments, re-reviews after discussion, numerical scores, and other recommendations as stipulated in law/regulations?
•	How are funding decisions made? To what extent are peer reviews used in those decisions (e.g., to establish rankings of grant/cooperative agreement applications to provide additional information for decisionmaking)?
•	What changes/reforms in the peer review system could be recommended to help improve the system?

Data Collection

To conduct the inquiry outlined above, a number of data collection strategies were undertaken. Interview protocols for applicants and reviewers required Office of Management and Budget (OMB) review. These interviews qualified as customer surveys under guidelines of the Paperwork Reduction Act and received an expedited OMB review. The following are the data collection strategies employed:

•	Review of OERI legislation, regulations, standards, and other documents related to peer review.
•	Conduct of initial open-ended interviews with key federal officials involved in FIS and center competitions to understand the overall review process and use of peer reviewers. Initial interviewees included key staff of the five institutes that have administered the two rounds of FIS competitions and nine center competitions held since the standards were adopted. This initial set of interviews helped us further refine our research questions, identify additional staff for further inquiry, and collect important documents for analysis.
•	Semistructured interviews with involved U.S. Department of Education (ED) staff on each of the research questions outlined above: appropriate uses of peer reviewers for FIS and center competitions, the peer reviewer selection process, links between the expertise of peer reviewers and that required for proposal reviews, the effectiveness and efficiency of the review process, the quality and usefulness of the reviews generated, the use of reviews in funding decisions, and recommendations for improving the system.
•	Semistructured interviews with a sample of reviewers on issues including links between their expertise and applications reviewed, the quality of instruction and training for the review process, the quality of the overall reviews, the quality and usefulness of review meetings or second-stage reviews in which they participated, and additional observations or recommendations.
•	Semistructured interviews with a sample of successful and unsuccessful applicants on issues including the quality of reviews and feedback received and additional observations or recommendations.
•	Semistructured interviews with appropriate staff of other ED offices and federal agencies selected for comparison, as well as representatives of educational research associations (e.g., the American Educational Research Association [AERA]), on issues including the appropriateness of OERI law/regulations, peer review practices in other offices or agencies, and recommendations.
•	Review of reviews performed on applications of sampled applicants.
•	Review of resumes of sampled reviewers.
•	A general literature review, with emphasis on the process used by ED offices and other federal agencies selected for comparison.

Sampling Plan

Because we did not have the resources needed to interview all applicants and peer reviewers involved in the FIS and center competitions, we conducted case studies of selected competitions. A total of 367 reviewers reviewed 1,179 applications for the FIS competitions in fiscal years 1996 and 1997. A total of 61 reviewers reviewed 47 applications for the center competitions during those same years.

For the FIS competitions, we studied 20 review panels across the 5 institutes and the 2 fiscal years. One panel from each of the five institutes was selected for more in-depth study because it was identified as successful by the institute director.¹ The remaining 15 panels were selected through a stratified random sample, ensuring that all institutes and both years would be represented. For each of the five "successful" panels, to the extent possible, we interviewed all three panelists, randomly selected two unsuccessful applicants, and interviewed all successful applicants whose applications had been reviewed by that panel.² For the remaining 15 panels we constructed a more limited picture, interviewing two randomly selected panel members and two applicants from each, again to the extent possible. Successful applicants associated with the panel were automatically included and randomly selected from the pool of successful applicants if there were more than two per panel. If there were not two successful applicants, randomly selected unsuccessful applicants were interviewed instead. This approach resulted in 34 interviews of applicants and 40 interviews of reviewers across the competitions. To the extent that they were available, we also reviewed the material associated with the randomly selected applicants and reviewers—38 applications, 100 reviews, and reviewers' resumes.

For the center competitions, we conducted two in-depth case studies—one center competition identified as highly successful by ED staff, and the other identified as problematic. To the extent possible, we interviewed three applicants (including the successful applicant) and three reviewers from each of these competitions. As with the FIS competitions, for each of the other seven center competitions we constructed a more limited picture. To the extent possible, we interviewed two randomly selected reviewers and one randomly selected unsuccessful applicant, as well as the successful applicant. This approach entailed 17 interviews of applicants and 14 interviews of reviewers across the center competitions. To the extent that they were available, we reviewed the materials associated with the randomly selected applicants and reviewers—12 applications, 41 reviews, and reviewers' resumes.

Data Analysis

Analysis of laws and regulations. We conducted an analysis of the key laws and regulations that govern OERI grant and cooperative agreement competitions. The analysis considered the requirements of the laws/regulations and their implications for conducting competitions. It also considered any inconsistencies, problematic elements, and conflicts among requirements.

Analysis of interview data. Because most of the information gathered for the study was based on semistructured interviews, the main analytic approach was qualitative data analysis. Data from interviews with applicants and reviewers was organized into tables by FIS panel and center competition. The data did not reveal enough consistency within panels in the assessment of review quality and reviewer expertise to warrant separate analyses. Thus the data was aggregated across cases and described in narrative form. This approach to analyzing qualitative data is widely used for aggregating and summarizing interview data.

Analysis of the match between applications and reviewers. To the extent they were available, we collected the resumes of the individuals who reviewed sampled applications, and analyzed the match between the substantive requirements of the applications and the background, experience, and subject matter expertise of the reviewers.

Analysis of application reviews. We developed criteria, based in part on the new standards, for assessing the quality of the written reviews. Measures of quality included thoroughness and detail in comments, discussion of substantive issues in the subject area of the application, and judgments that were amply and expertly justified.

Analysis of review process of six panels nominated by OERI staff. In addition to our overall picture of the competitions, we examined the review process of six panels in greater depth. This examination focused on four fiscal year 1997 FIS panels nominated by institute staff as particularly successful and two center competition panels—one nominated as successful and one as problematic.

U.S. Department of Education staff interviews. The staff who were involved in administering and overseeing the competitions and staff from several professional associations were interviewed. Relevant comments and insights have been incorporated in this analysis.

Review of peer review in other offices and agencies.³ We also examined the peer review process conducted in other offices within the U.S. Department of Education; federal agencies; and a private foundation, the Spencer Foundation, that primarily funds education research. The other offices within the Department included OERI's Office of Reform Assistance and Dissemination (specifically the Technology Challenge Grants), the Division of Innovation and Development in the Office of Special Education Programs, and the Fund for the Improvement of Postsecondary Education in the Office of Postsecondary Education. Other federal agencies included the National Institutes of Health, the National Science Foundation, and the National Endowment for the Humanities.

Organization of this Report

Chapter 2 of this report reviews the specific requirements embodied in the new standards. Chapter 3 describes the FIS and center competitions conducted in fiscal years 1996 and 1997. Chapter 4 presents the study findings regarding the fit between reviewers and the applications they reviewed, the quality of the reviews performed, and the review process of the six panels examined in depth. Chapter 5 presents conclusions and recommendations of the study. The appendices include background material for the study: appendix A is a copy of the OERI/NERPPB standards governing the conduct and evaluation of grants and cooperative agreements; appendix B is the Technical Review Form; and appendix C provides the evaluation criteria and specific factors used to rate applications for FIS and center competitions.

-###-

[ Executive Summary ]

[ The Nature of Standards ]