A r c h i v e d  I n f o r m a t i o n

III. Description of the Competitions

Field-Initiated Studies Research Grant Program,

Fiscal Years 1996 and 1997

This section describes the procedures used for the field-initiated studies (FIS) competitions in fiscal years (FY) 1996 and 1997. It draws on information presented in the technical review plans and slate recommendations for the competitions, as well as from U.S. Department of Education (ED) staff interviews.

Background

Under Section 931 of Title IX of the Educational Research, Development, and Improvement Act of 1994 (Public Law 103_227), the Office of Educational Research and Improvement (OERI) held a discretionary grant competition for FIS grants. "Standards for the Conduct and Evaluation of Activities Carried Out by the Office of Educational Research and Improvement" (34 Code of Federal Regulations [CFR] Part 700), provides program criteria for applicants' proposals and guidance for the peer review process.

In general, each institute may fund only applications for studies that are within its legislative mission. In the application package, applicants were instructed that they were to apply to one of five institutes. Applicants were also invited to address one of the priorities cited in OERI's Research Priorities Plan. However, it was clear from the application package that addressing these priorities was not mandatory. Specifically, the text states: "Applicants that address one of these priorities will not receive competitive or absolute preference over other applicants."

Generally, grant funds support research projects averaging $150,000 per year and from 1 to 3 years in duration. In FY 1996, 45 awards were made from among a total of 647 applications. In FY 1997, 30 awards were made from among a total of 532 applications.

Compilation of Reviewer Rosters

In accordance with the OERI/NERPPB standards, each institute's work team,¹ in collaboration with the institute directors, prepared two lists of reviewers prior to the start of the panel reviews—a primary list and a list of alternates. If a reviewer from the primary list could not serve, another reviewer from the approved list of alternates was to be invited. The lists of primary and alternate reviewers were submitted to the institute directors. In FY 1997, about 30 percent of those on the primary list had served as reviewers in FY 1996, with the remainder being new reviewers. The list of alternate reviewers included available current FIS project directors.

According to OERI staff, reviewers were selected who had subject area expertise in the areas in which the institutes anticipated receiving proposals, according to their mission statements and the topic areas included in the Research Priorities Plan. Some institutes waited until the applications had been received and they knew the application topics before constructing the review panels. Others formed preliminary panels based on anticipated application topics. The institute directors met to ensure that there would be no overlap in reviewers across institutes.

A file containing information about potential reviewers is maintained in the program office; it shows the reviewers' qualifications and availability to participate in the review process. The availability status of reviewers is updated as staff begin to recruit reviewers for each competition. In FY 1996, the Department published a request for reviewers in the Federal Register and in the OERI Bulletin. The need for qualified reviewers is announced to professional associations at annual meetings, through departmental announcements, and through direct recruitment by program staff.

Screening of Applications

The U.S. Department of Education staff screened applications for eligibility and conformance with the regulations governing the program, and reviewed them for completeness. Applications were rejected if they did not meet the statutory eligibility requirements under the authorizing statute or Education Department General Administrative Regulations (EDGAR); if they were not complete; if the applicant was not eligible; or if there were no funds available within OERI to address the subject of the application. In FY 1996, 72 applications were deemed ineligible; in FY 1997 the number of ineligible applications was 56.

Conflicts of Interest, Assurances, and Waivers

Reviewers were required to sign one of two forms, depending on whether they were to be paid for their services.² Reviewers who reviewed and evaluated applications at no cost to the government were required to sign a Gratuitous Services Agreement form. Any reviewers who were to be paid travel expenses, per diem, and an honorarium were required to sign an Agreement for Grant Application Reviewers Who Receive Compensation form, as well as a Drug-Free Workplace Certification form.

These forms also contain a "Conflict of Interest" section. Conflict of interest is defined in 34 CFR Part 700, "Standards for the Conduct and Evaluation of Activities Carried Out by Office of Educational Research and Improvement," Section 700.13. Each institute was to submit a request for blanket waivers of conflict of interest when a significant majority of reviewers were affiliated with institutions that had submitted applications, but were not directly associated with those applications. Individual blanket waivers were provided in FY 1996. In FY 1997, to cover cases in which reviewers were employees of a university but did not have direct involvement with applications from that university, a joint request (covering all five institutes) for a blanket waiver of conflict of interest was signed by Ramon Cortines, Acting Assistant Secretary, on June 16, 1997.

At the beginning of the review process in FY 1996, a list of assigned applications was sent to each reviewer to allow identification of any conflict of interest as defined by instructions. The U.S. Department of Education staff contacted reviewers to confirm their receipt of this list and to ensure that there were in fact no conflicts of interest. In FY 1997, an outside contractor sent reviewers the applications. Reviewers were told to contact the contractor to confirm receipt of the applications, and to contact their panel's chair immediately if there were a conflict of interest or other problem. (In both years, during recruitment, staff also asked reviewers to inform them about any potential conflicts of interest.) Reviewers were also asked to sign the "Conflict of Interest" section of the relevant form, stating that no conflict existed and that if one arose, they were obligated to inform the program official and would do so. If a reviewer had a conflict of interest involving any application in the competition, that reviewer was not to read at all unless a waiver had been signed by the Office of the General Counsel.

Orientation Instructions for Reviewers

FY 1996. An orientation package was sent to reviewers. It included a description of the program purpose and key review dates, a copy of the application package, the Technical Review Form (both hard copy and diskette), and a copy of the Horace Mann Learning Center workbook for reviewers. A second package was subsequently sent out, containing instructions for reviewers; a list of applications assigned to panel members; and a Gratuitous Services Agreement form, or an Agreement for Grant Application Reviewers Who Receive Compensation and Drug-Free Workplace Certification form.

FY 1997. An independent contractor was hired to assist with the logistics for the competition. The contractor sent a preliminary letter to reviewers to confirm the dates of the panel meetings and explain hotel and travel arrangements to out-of-towners and locals. Approximately 3 weeks before the review panels were convened, each confirmed reviewer received a package containing a thank-you letter; reviewer instructions (including travel and hotel specifics); a Technical Review Form and disk (same for all institutes, but with institute-specific headings); a copy of the Horace Mann Learning Center workbook for reviewers; a list of assigned applications and panel members; assigned applications; and a Gratuitous Services Agreement form, or an Agreement for Grant Application Reviewers Who Receive Compensation and Drug-Free Workplace Certification form.

Basis for Assigning Applications to Reviewers

Every application was to be read by a panel of three reviewers. Applications were to be assigned to panels according to the fit between panel expertise and project topics. According to ED staff, to the extent possible, the three reviewers on the panel were also categorized as a specialist in the subject area, a generalist, and a methodologist. To avoid conflict of interest, reviewers were not given applications from institutions or organizations with which they were affiliated or with which they had any financial or working relationship (past or present). Across the institutes, there were 218 reviewers in FY 1996 and 147 in FY 1997.

On-Site Orientation (FY 1997 only)

On the morning of the first day of panel reviews in FY 1997, ED staff provided an orientation for reviewers. Staff from various institutes discussed the review process and showed a film on the peer review process.

Review Process

Reviewers had access only to the applications assigned to their panels, and were not given any information about applications reviewed by other panels. In most cases, panels were expected to read at least 10 applications. In FY 1996, reviewers rated applications off site and conferred via telephone only if there were significant discrepancies among the reviewers' scores. In FY 1997, reviewers were asked to complete their preliminary review of applications by mail and then met in Washington, DC, to complete the evaluations as a panel. Reviewers met as panels of three, and each panel was assigned a chair who was responsible for facilitating the timely discussion of applications. An ED staff member was assigned to each panel to facilitate the panel discussion and answer reviewers' questions. Reviewers discussed their reviews, generally one application at a time, and following the discussion were allowed to adjust their scores and revise their comments accordingly. They were instructed to make score adjustments independently of other reviewers. At the conclusion of the review process, each panelist was to place each application in one of three categories: highly recommended for funding, recommended for funding, or not recommended for funding.

Ranking of Applications

Following completion of the panel review, the program office entered the scores of each reviewer for each application on a panel summary form. Average raw scores were then computed from the reviewers' individual raw scores. These raw scores were entered into each institute's database to establish an initial rank order. In cases in which application scores were standardized, the Department's Application Control Center used these raw scores for that purpose.

Standardizing of Scores

In FY 1996, the Policy Institute used the Department's discretionary grant application score standardization process; all other institutes used raw scores. In making 1997 FIS grant awards, four of the five institutes used the score standardization process; the exception was the National Institute on Educational Governance, Finance, Policymaking, and Management.

Making of Grant Awards

According to the Department staff, their funding recommendations were consistent with the results of the panels' technical reviews, and the highest-rated applications were selected for funding, with one exception. In FY 1996, staff deviated from the panelists' rank ordering because the work proposed in one application duplicated work carried out by the applicant with other OERI funds. In two other instances, staff selected one of two equally rated applications (both could not be funded given available resources) based on their judgment as to which would contribute most to the field.

Funding slates were prepared that contained the funding recommendations of the collaborative work team, the institute directors, and representatives of OERI's Office of the Assistant Secretary, along with the rank-ordered list of all applications reviewed. The institute directors and team members signed the slate of recommendations. After the awards had been announced, unsuccessful applicants were sent regret letters signed by the work team leader, along with copies of the review forms.

Program office staff were to send original unfunded applications and related documents to the Federal Records Center within 90 days after awards had been made and unsuccessful applicants had been notified. These files are to be retained by the Federal Records Center for 3 years. Any other copies of the applications and related documents are to be destroyed in a manner that preserves their confidentiality.

National Educational Research and

Development Center Program Competitions

Overview

The Educational Research, Development, and Improvement Act of 1994 required OERI to continue to fund centers, using at least one-third of the funds available to each institute in any fiscal year for this purpose. Prior to the FY 1996 competition, the first to be conducted under the new OERI/NERPPB standards, ED staff met to resolve three key issues: the number of centers to fund; funding levels for each; and topic areas to be addressed by the new centers.

The Department staff developed priorities for seven new research centers. They did so through a lengthy process that involved notices published in the Federal Register for public comment, OERI staff work group papers, meetings with national educational associations and organizations, public meetings in five states, expert panel reviews, NERPPB discussions, and internal ED staff reviews. The seven priorities were as follows: Priority 1, Enhancing Young Children's Development and Learning; Priority 2, Improving Student Learning and Achievement; Priority 3, Ensuring Effective Teaching; Priority 4, Strengthening Schools; Priority 5, Meeting the Needs of a Diverse Population; Priority 6, Promoting Learning Beyond the Classroom; and Priority 7, Improving Adult Learning and Literacy. Each center was to conduct a coherent, sustained program of research and development, and address problems and issues of national significance in its individual priority area.

In FY 1996, seven distinct but coordinated competitions were held to fund centers in the above seven priority areas.³ The program competition was announced in the Federal Register on September 14, 1995, and the closing date for receipt of applications was December 15, 1996. Government furloughs, fear of congressional recision of center funding, and a snowstorm in January resulted in later closing dates for receipt of applications and caused delays in the review process for several of the centers.

In FY 1997, two distinct competitions were carried out—Research on Policy and Teaching Excellence and Research to Improve Children's Early Reading. The first was announced in the Federal Register on April 11, 1997, with a closing date of June 6, 1997; the second was announced on March 26, 1997, with a closing date of May 28, 1997.

In FY 1996, 8 center awards were made from a total of 38 applications. In FY 97, 2 center awards were made from a total of 9 applications.

Composition of Review Panels

In FY 1996, OERI staff established review panels consisting of approximately 5_10 peer reviewers for each of the competitions, depending on the expertise needed. Peer reviewers were to possess the qualifications specified by the standards (see chapter 2). Names of potential reviewers were generated by database searches and staff recommendations; the Department staff contacted grantees and examined past lists of reviewers. Nonfederal reviewers were paid an honorarium of $750.

In FY 1997, the Policy Institute prepared a primary list of reviewers and a list of alternates, selected from a comprehensive file of reviewers maintained in the program office and from work team recommendations. For the competition to award the Center on Early Reading, five potential reviewers were identified by the Achievement Institute's work team and two by Early Childhood Institute staff.

Conflicts of Interest, Assurances, and Waivers

As with the FIS panels, reviewers were required to sign one of two forms, depending on whether they were to be paid for their services.

Reviewers were informed early and often about the requirements regarding conflict of interest. They were asked to notify OERI staff immediately if their institution was planning to apply for a center award for which they had been asked to serve as a reviewer, or if they or a family member would benefit from the outcome of the competition. To assist in identifying any conflicts of interest, each reviewer was asked to examine the list of applications assigned to his or her panel prior to the beginning of the review, and OERI staff reviewed applications before mailing them to reviewers.

Screening of Applications

As with FIS applications, ED staff screened applications for eligibility and conformance with the regulations governing the program, and reviewed applications for completeness.

Technical Review Form

The application Technical Review form was used to review applications (see appendix B).

Orientation for the Panels

Prior to each panel meeting, peer reviewers received by mail a general letter of instructions, a set of reviewing instructions, a copy of the application package, copies of the assigned applications, and the application Technical Review Forms (both hardcopy and electronic versions). The letter for reviewers was uniform, except for references to the specific priority area for each panel. The letters and materials were sent out under the signature of the director of the institute responsible for the panel. Institute staff contacted peer reviewers by telephone and arranged for one reviewer on each of the panels to serve as chair. In some cases institute staff assigned responsibility for presenting and discussing given applications to specific reviewers on a panel.

On the first morning of the panel meetings, there was an orientation session that included a representative from the Department's Grants and Contracts Office. During this meeting, OERI staff briefly reviewed the selection criteria and stressed the need for full documentation of all scores. Reviewers were reminded to evaluate each application independently of each other, evaluate the application based on an assessment of its quality according to the evaluation criteria and the weights assigned to those criteria, and support the rating assigned to the application with concise written comments based on an analysis of the strengths and weaknesses of the application with respect to each of the applicable evaluation criteria.

Conduct of the Review Panel

A one-tier review was conducted in three phases. First, all reviewers reviewed and scored all applications received for the priority area they were addressing according to the established selection criteria. Second, reviewers on each panel met as a group in Washington, DC, and discussed applications assigned to their panel. At the beginning of the meeting, preliminary scores were posted for all applications. All of the applications were discussed, and discussion was concluded on those applications whose preliminary scores, ratings, and evaluations were very weak. The panelists again reviewed the remaining applications. Next, each reviewer had the opportunity to independently reevaluate and re-rate any of the applications and make changes to the written comments as appropriate. The applications were then discussed to share information, not to seek consensus. Third, the rank order of the applications—based solely on the reviewers' scores—was calculated and posted. In some cases, reviewers identified critical issues for the highest-ranking applicants. These issues were used by OERI staff to formulate questions for clarification that were transmitted to applicants for response before awards were made.

U.S. Department of Education Staff Role

OERI staff provided an orientation for reviewers prior to the review, attended panel meetings, answered any questions that arose, and examined review forms for accuracy and completeness. They also prepared clarification/negotiation questions for potential awardees. In addition, they prepared funding recommendation memoranda for the Assistant Secretary's approval.

Selection of Applications for Award

The Assistant Secretary selected applications for award, considering the applicants' ranking by the peer reviewers, recommendations of the peer reviewers with regard to funding or no funding, information concerning applicants' performance and use of funds under a previous federal award, the amount of funds available for a competition, and any other information relevant to a priority area or regulatory requirement applicable to the selection of applications for new center awards. In no case did the Assistant Secretary's selection deviate from a panel's rank ordering.

Applications Not Selected for Funding

Applicants whose proposals were not selected for funding received a letter following the congressional notification of funded applicants and the award of the cooperative agreements.

Specifics of the Competitions, FY 1996

Priority #1. The National Institute on Early Childhood Development and Education supports the Research and Development Center on Enhancing Young Children's Development and Learning, University of North Carolina, at $2.75 million per year for 5 years. For this competition, seven applications were submitted for consideration by a panel of five reviewers.

Priority #2. The National Institute on Student Achievement, Curriculum, and Assessment supports the National Research and Development Center on Improving Student Learning and Achievement in English, the University of Albany, State University of New York, at $2.5 million a year for 5 years. The National Institute on Student Achievement, Curriculum, and Assessment also supports the National Research and Development Center on Achievement in School Mathematics and Science at the Wisconsin Center for Educational Research, University of Wisconsin, for $2.5 million a year for 5 years. For this competition, 6 applications were submitted, 10 reviewers participated, and 2 awards (referred to above) were made.

Priority #3. The National Institute on Student Achievement, Curriculum, and Assessment supports the National Research and Development Center for Student Assessment and Educational Accountability, the University of California, Los Angeles, at $2.8 million per year for 5 years. For this competition, two applications were reviewed by a panel of seven reviewers.

Priority #4. The National Institute on the Education of At-Risk Students supports the National Center for Research on Education, Diversity, and Excellence, the University of California, Santa Cruz, at $4 million per year for 5 years. For this competition, eight applications were submitted for review, and five external reviewers participated.

Priority #5. The National Institute on Educational Governance, Finance, Policymaking, and Management supports the Consortium for Policy Research in Education, the University of Pennsylvania, at $2.8 million per year for 5 years. For this competition, five organizations submitted applications that addressed the priority "Increasing the Effectiveness of State and Local Education Reform Efforts," and six reviewers evaluated applications.

Priority #6. The National Institute on Postsecondary Education, Libraries, and Lifelong Learning supports the National Center for Postsecondary Improvement, Stanford University, at $2.5 million per year for 5 years. For this competition, six proposals were submitted and were reviewed by a panel of five individuals.

Priority #7. The National Institute on Postsecondary Education, Libraries, and Lifelong Learning supports the National Center for the Study of Adult Learning and Literacy, Harvard University/World Education, at $2.5 million per year for 5 years. For this competition, eight proposals were submitted, and a panel of nine reviewers met to discuss the applications.

Specifics of the Competitions, FY 1997

National Center on Early Reading. The Student Achievement Institute and the Early Childhood Institute support the Center for the Improvement of Early Reading, the University of Michigan, at $2.3 million per year for 5 years. Five applications were submitted for review, and seven external reviewers served on the panel.

National Research Center on Policy and Teaching Excellence. The Policy Institute supports the National Center on Policy and Teaching Excellence, the University of Washington (in collaboration with four other universities), at $1.5 million for 5 years. Four applications were submitted for review, and seven external reviewers served on the panel.

-###-

[ The Nature of Standards ]

[ Findings ]