A r c h i v e d  I n f o r m a t i o n

II. The Nature of the Standards

Before examining the implementation of the new OERI/NERPPB standards for conducting application reviews, it is important to understand what those standards require. This chapter describes each important element in the standards and reflects briefly on associated implementation issues that have arisen. The elements described include the qualifications and selection of peer reviewers who review applications for grants and cooperative agreements; the rules for conducting the competitions; and the process for evaluation, ranking, and selection of award recipients.

Qualifications of Peer Reviewers

The standards require that individuals selected as peer reviewers have the following qualifications:

•	Demonstrated expertise, including training and experience, in the subject area of the competition.
•	In-depth knowledge of policy or practice in education.
•	In-depth knowledge of theoretical perspectives or methodological approaches in the subject area of the competition.

Because there has been considerable confusion about who can serve as a peer reviewer, these requirements are quoted as they appear in the regulations (Federal Register, September 14, 1995:47811). All three conditions would appear to apply to all individuals who review applications; there is nothing in the standards to suggest that it is sufficient for the reviewer panels to reflect the three qualifications collectively.

The wording of the first qualification indicates that determining the subject area of a competition is critical to determining who is qualified to serve as a peer reviewer. A major issue that has arisen in practice is determining whether the subject area means research about the subject area. If the competition is the field-initiated studies (FIS) Educational Research Grant Program, the subject area would appear to be research within the purview of the institutes participating in the program (e.g., early childhood education, education of at-risk children). The second qualification, in-depth knowledge of policy or practice, seeks to ensure that, in addition to expertise in the subject area of the competition, all reviewers have some grounding in the practice of education. The importance of research expertise would appear to be reinforced by the third qualification—in-depth knowledge of theoretical perspectives or methodological approaches in the subject area of the competition. This requirement goes further than simply requiring expertise in a field; it specifies two important components of expertise and indicates that all reviewers must have at least one of them. All reviewers must come to the table with in-depth knowledge of theory or methods in the area of the competition. One of those components—in-depth knowledge of methods—is a subset of research expertise. The other—in-depth knowledge of theory—might exist apart from research expertise, but the likelihood of an individual having in-depth theoretical understanding without a research background would be small.

There is little doubt that considerable effort is required to find individuals who meet all three qualifications in the standards. Nonetheless, the standards should be, and indeed appear to be, an effort to raise the bar, ensuring a high-quality group of peer reviewers who will be able to provide expert reviews and lend insight and stature to the peer review efforts of OERI. By requiring that all members of a panel meet all three qualifications, the standards clearly indicate that creating a panel is not to be a mix-and-match effort, with one reviewer representing the subject area, one practice, and one methods or theory, as some have described it. Rather, the standards indicate that all peer reviewers should possess all three qualifications.

Selection of Peer Reviewers

In addition to the individual qualifications for peer reviewers, the standards require that OERI (the Secretary) select "to the extent feasible. . .peer reviewers for each competition who represent a broad range of perspectives." This requirement is not further elaborated in the discussion of the rules, so it is not immediately apparent how it should be implemented. In the context of research applications, a broad range of perspectives could mean reviewers who differ with regard to disciplinary background within a broad area such as policy or at-risk children (e.g., education psychologists, ethnographers, sociologists), theoretical orientation, methodological approach, research role (e.g., academic researchers, contract researchers, voluntary sector researchers), or any number of other characteristics.

What is clear from this requirement is the desire to avoid a situation in which all reviewers of an application share the same viewpoint on what is legitimate, appropriate, or feasible in a particular research field.

Conduct of the Competitions

The standards also prescribe certain aspects of the competition, including when peer review is to be used, the minimum numbers of reviewers of an application, reviewer obligations, the review process, and evaluation criteria. First, the rules specify that peer review is to be used for the review and evaluation of all applications for grants and cooperative agreements that exceed $100,000.¹ Later in the regulations, however, the prescription appears to vary somewhat, with the language noting that fewer than three reviewers may be used for grant and cooperative agreement awards under $50,000 "if the Secretary determines that adequate peer review can be obtained using fewer reviewers." This clause appears to extend downward the purview of peer review, but it may be that peer review is discretionary at this funding level. Nonetheless, the clause adds the important criterion of at least three reviewers for grant or cooperative agreement awards above $50,000. In addition, awards above $1,000,000 require at least five peer reviewers.

The standards also specify a set of broad evaluation criteria and specific factors from which OERI may select specific items appropriate to each grant or cooperative agreement competition. The broad criteria include national significance, the quality of the project design, the quality and potential contributions of personnel, the adequacy of resources, and the quality of the management plan. The specific factors offer a wide range of options; some are oriented to research competitions, while others are more reflective of demonstrations or program grants. The rules allow for complete discretion with respect to which of these broad criteria and specific factors are used in any competition.

The review process is spelled out in some detail. First, reviewers "must be given a number of applications to evaluate," although no specific number is cited. This requirement may have been inserted to give the reviewers a better perspective on the quality of applications than would be provided by a single review. The rules here appear to acknowledge that ultimate decisions to award grants are based on relative rankings of applications.

Reviewers are instructed to "independently evaluate" and rate (i.e., score) each application. Evaluations and ratings are to be based on the applicable evaluation criteria and weights assigned to each criterion. The evaluations are to be accompanied by "concise written comments based on the reviewer's analysis of the strengths and weaknesses of the application with respect to each of the applicable evaluation criteria." After the independent evaluation/rating, reviewers who evaluated "a common set" of applications are to convene and discuss the strengths and weaknesses of those applications. "Each reviewer may then independently reevaluate and re-rate an application with appropriate changes made to the written comments." After this process has been completed, "reviewers shall independently place each application in one of three categories, either 'highly recommended for funding,' 'recommended for funding,' or 'not recommended for funding.'"

The emphasis throughout the discussion of the review process in the standards is on independent judgment by reviewers. First, the evaluation of applications is to take place prior to meetings among reviewers. After the discussion of application strengths and weaknesses, each reviewer is to make an independent reassessment and revise his or her scores and written comments. Finally, after the review of all applications has been completed, reviewers are to independently assign each application to one of the above three categories (i.e., highly recommended, recommended, or not recommended). In essence, the discussion process is an opportunity for the reviewers to gain additional information, perhaps a sense of different viewpoints and perspectives, on the proposed research. Given the emphasis on individual decisionmaking, the review is clearly not the point at which collective decisions are made on application ratings or desirability for funding.

Evaluating, Ranking, and Selection

of Award Recipients

Once the peer review process has been completed, several important steps must occur before awards are made. First, "the Secretary prepares a rank order of the applications based solely on the peer reviewers' ratings." Then, "the Secretary determines the order in which applications will be selected for grant and cooperative agreement awards." The standards require the Secretary to consider a wide array of information, including an applicant's ranking, recommendations of the peer reviewers with regard to funding or not funding, information concerning an applicant's performance and use of funds under a previous federal award, amount of funds available for the competition, and any other information relevant to a priority or other statutory or regulatory requirement applicable to the selection of applications for new awards. Clearly, the Secretary has considerable latitude with respect to making awards once the peer review process has been completed.

There are several important features of this prescription for making awards. First, peer reviewer rankings are important, but they are by no means the only consideration in making awards. Second, previous performance may be considered, but the rules are not explicit about who makes such an assessment or how. Third, limited funding may be considered, and could presumably play several roles. It could result in passing over a single expensive project, but it could also result in deciding to support more (or fewer) projects overall, shifting awards as a whole to a different set of projects than might have been funded had rankings alone been considered. Finally, statutory priorities or other statutory requirements can be invoked to support applicants that may not have received awards based solely on rankings.

Procedures Not Specified by the Standards

It is also important to note what is not covered by the standards. Among those items are: priorities that may be identified in program announcements or application packages, but are not found in statutes or regulations; modification of reviewer rankings through statistical manipulations in order to standardize the rankings; the efficacy of multistage reviews; and use of standing panels.

Within OERI, priorities that do not carry the weight of law or regulation are often cited in application packages. While the text of the package may point out that the priorities are nonbonding, it sometimes remains unclear how reviewers and others should view those priorities in deciding on application strengths/weaknesses, rankings, and awards.

With respect to standardization of scores, some grant competitions and some institutes have elected to use ED's standardization process in creating their rankings, while others have not. This process is designed to correct for possible bias introduced by different reviewers' approaches to assigning raw scores (i.e., some reviewers tend to score high, while others tend to score low). The standardization process is based on certain assumptions about the distribution of applications and the behavior of reviewers. According to the Department's information, the standardization process assumes:

that the varying quality of applications in the entire pool of applications is normally distributed, (i.e., a similar number of good, average, and poor applications; the applications distributed to each panel are normally distributed; all panels have the same training and direction; and any resulting deviation is due to reader bias).

Multistage reviews—which usually entail a first-stage review by peers, followed by a second-stage review by peers and/or a combination of peers and other stakeholders—are commonly used by other federal research agencies, often to separate judgments about research quality from those about policy significance. Standing panels comprising individuals who meet the standards for qualified reviewers and serve specified terms, generally for several consecutive years, are also used by other agencies. While such reviews and panels are not addressed in the standards, the Secretary's discretionary powers in making grant awards (especially with regard to obtaining other information on priorities or requirements) would appear to make second-stage reviews and standing panels possible if they were undertaken to assist the Secretary in decisionmaking, and if the applicants were aware that this process was being employed. Second-stage reviews and standing panels are so widely used by other federal research agencies (such as the National Institutes of Health and the National Science Foundation) that it is certainly worth considering further whether they are allowed by the OERI/NERPPB standards.

[ Overview ]

[ Description of the Competitions ]