A r c h i v e d  I n f o r m a t i o n

Changing Federal Strategies for Supporting Educational Research, Development, and Statistics - September 1998

VII. The "New" OERI

The reauthorization of OERI in March 1994 significantly altered the structure as well as some of the operating principles of the agency. Combined with the return of the Democrats to the White House after a hiatus of more than a decade, many educators and policy makers hoped for a more harmonious partnership between OERI and Congress which might result in some major improvements. Yet often unanticipated and unintended changes at the agency weakened its ability to provide high quality educational research, development, and statistics.

The Transition in OERI

Emerson Elliott was named as the acting Assistant Secretary of OERI on January 22, 1993. Elliott was the Commissioner of Education Statistics at the National Center for Education Statistics and was credited with having transformed the organization into a highly effective and widely admired federal statistical agency.⁽¹¹¹⁾ While the few strictly political appointees in OERI from the Bush Administration were told to resign (the so-called Schedule C appointees), Elliott asked everyone else to remain at their posts and to continue to function as before.⁽¹¹²⁾

The White House nominated Sharon Robinson as the new Assistant Secretary of OERI.⁽¹¹³⁾ Robinson had earned a doctorate in 1979 from the University of Kentucky in educational administration, served as a middle school and high school classroom teacher, and was the Director of NEA's Instructional and Professional Division from 1980 through 1989. Since January 1990 she had been the Director of the National Center for Innovation-NEA's research and development unit. Robinson was confirmed by the U.S. Senate in late June 1993.⁽¹¹⁴⁾

Unlike Diane Ravitch, her predecessor, Robinson was not considered a major researcher or a distinguished scholar, but she was deeply committed to increasing the utilization of educational research in the classroom and improving teacher professional training. Robinson was particularly interested in closer collaborations between researchers and classroom teachers and emphasized the value of teamwork. She downplayed expanding and revitalizing OERI's research staff, but stressed the need for developing working partnerships between researchers and teachers. Providing equal public educational opportunities for all children was one of her primary goals - especially helping disadvantaged minority students to obtain adequate resources and assistance. She was a very dynamic and thoughtful speaker and received a rare standing ovation from the OERI staff at her first general agency meeting.⁽¹¹⁵⁾

As we have discussed earlier, disagreements over the nature and powers of the proposed OERI policy board had stalled the reauthorization of OERI in 1991 and 1992.⁽¹¹⁶⁾ With the arrival of the new administration, there was considerable optimism that OERI might be reauthorized quickly, but that did not occur in 1993.⁽¹¹⁷⁾ A compromise was finally reached the following year. Rather than developing and controlling the long-term OERI research agenda, the policy board now was authorized to work together with the Assistant Secretary to draft that plan. Having settled their few differences, OERI was easily reauthorized as part of Goals 2000: Educate America Act on March 31, 1994 (as Title IX of P.L. 103 - 27).⁽¹¹⁸⁾

The law called for the establishment of five National Research Institutes: the National Institute on Early Childhood Development and Education; the National Institute on Student Achievement, Curriculum and Assessment; the National Institute on Education of At-Risk Children; the National Institute on Postsecondary Education, Libraries, and Lifelong Education; and the National Institute on Educational Governance, Finance, Policymaking, and Management. The research institutes together were authorized at $100 million for FY 1996; at least 20 percent of those funds had to be allocated to field initiated research and at least one-third to national research centers. The law created a 15-member National Educational Research Policy and Priorities Board (NERPPB). The legislation also called for the creation and use of high quality standards for the conduct and evaluation of OERI research activities. Finally, the law mandated the establishment of the Office of Reform Assistance and Dissemination (ORAD) which would include institutions and activities such as the regional educational laboratories, the Educational Resources Information Clearinghouses (ERIC), and the National Diffusion Network (NDN).

The reauthorization of OERI was welcomed by most educators, policy makers, and researchers. Despite the earlier battles over the nature and powers of the National Educational Research Policy and Priorities Board, most commentators were satisfied with the eventual compromise. And almost everyone thought that the restructuring of the OERI was a step in the right direction-though some expressed concern about both the manner and the pace of implementing the new legislation.⁽¹¹⁹⁾

Funding Increases and Staff Reductions

In an earlier section of this essay, the dramatic increases in overall OERI funding during the Bush Administration were discussed. OERI funding continued to grow during the Clinton Administration - though at a considerably slower pace than in the previous 4 years. Overall funding rose from $286.2 million in FY 1993 to $398.1 million in FY 1997-a sizable 39 percent increase during those 4 years (the rate of increase in the previous four years had been a p phenomenal 266 percent). Even in constant 1996 dollars, the rise in overall OERI spending from FY 1993 to FY 1997 was a substantial 25 percent (but still much less than the 214 percent constant-dollar increase from FY 1989 to FY 1993).

The amount of money spent on the more traditional OERI programs (NCES, the centers and labs, field-initiated research, and ERIC) rose by only 25.3 percent from FY 1993 to FY 1997 while money for the other, more recently added functions increased by 54.0 percent. As a result, the proportion of OERI funds allocated for the more traditional programs continued to decline (98.7 percent in FY 1989; 52.8 percent in FY 1993; and 47.6 percent in FY 1997).⁽¹²⁰⁾

How one interprets the great expansion in OERI's nontraditional funding since FY 1989 depends in part on what one thinks about the current state of educational research and development. If one really believes that we already have a "treasure chest" of well-researched information that is simply awaiting broader dissemination, then the direction of the recent changes in relative funding at OERI are welcome. But if one thinks that we still lack adequate research and development to provide much guidance for educators and policy makers, then the increasing focus on educational technology and dissemination might seem somewhat premature and misplaced. For example, in FY 1997 approximately the same amount of money was being spent by OERI on new educational technology projects as was spent on the traditional centers, labs, field-initiated research, and ERIC combined. Given the recent sizable staff reductions, one might question the wisdom of a research-focused agency embarking on such new and large-scale initiatives - especially since many of them do not involve real research or development activities.⁽¹²¹⁾

One of the most neglected subjects has been the recent OERI staff decreases and their impact on the functioning of the agency. There was a sizable reduction in OERI employees during the early years of the Reagan Administration followed by a small recovery in staff from 425 in FY 1988 to 448 in FY 1992 (a 5.4 percent increase).⁽¹²²⁾

Staffing at the U.S. Department of Education during the Clinton Administration has been pulled in opposite directions. On the one hand, Congress expanded federal education funding and authorized the Administration to staff new or existing initiatives such as the Direct Federal Student Loan Program. These developments increased the need for additional staff. On the other hand, the reinvention of the federal government was predicated in large part upon delivering services more efficiently so that the overall size of the federal government staff could be reduced. The net result in practice has been a small decrease in staff full-time equivalents (FTEs) at the Department from 4859 in FY 1992 to 4655 in FY 1996 (a 4.2 percent decrease).

But if the overall size of the Department staff has remained relatively stable, there have been sizable and largely unnoticed shifts among the various agencies. OERI was one of the agencies which lost a large proportion of its staff - from 448 in FY 1992 to 358 in FY 1996 (a decrease of 20.2 percent). Moreover, the staff reductions did not end. By March 1997 OERI lost another 20 staff members (down to 338 FTEs). Compared with the situation when Assistant Secretary Ravitch left office in January 1993, the OERI staff had been reduced by approximately 25 percent.

Another perspective on the potential problems caused by the reductions in the staff is to look at changes in the level of federal expenditures per FTE in OERI. While this is a complex ratio and by no means an entirely accurate indication of the amount of actual work facing the staff, this crude index does provide us with another useful perspective on the changes over time. In the early 1980s each staff member at OERI represented in constant 1996 dollars approximately $170,000-$200,000 worth of activity. Even though the Reagan Administration made major cuts in staff, they were matched by reductions in agency spending so that the burden of work from this perspective remained roughly the same. During the Bush Administration, however, the large increases in OERI funding were only accompanied by modest staff increases so that by FY 1992 each individual in OERI accounted for approximately $975,000 in activities (in constant 1996 dollars) - nearly 4 or 5 times as much as earlier. With the continued increases in real expenditures at OERI as well as the one-quarter reduction in staff, each agency employee now represents $1.5 million (in constant 1996 dollars).⁽¹²³⁾ Thus, as the federal expenditures in real dollars in OERI have risen rapidly in the late 1980s and early 1990s, the availability of staff to oversee them has dropped dramatically.

At the same time that the permanent OERI staff was reduced, another major shift in staffing occurred without much notice from within or outside the agency-the elimination of excepted service hiring. When NIE was created in the early 1970s, one of the most useful procedures granted to the agency was the option to hire outstanding experts outside of the regular civil service system. These individuals could be employed for 3-year terms (renewable one time) to help with a particular problem. Critics charged that excepted service personnel were sometimes hired to circumvent the regular system or that the system was abused by politicians in order to hire unqualified friends. But overall, excepted service continued to be used despite periodic challenges to the actual practices.⁽¹²⁴⁾

In recent years, excepted service has fallen into almost total disuse at OERI. Congress was persuaded in 1994 to set much more stringent criteria for hiring anyone on the excepted service rolls. The new law still allows the hiring of scientific or technical employees for a 3-year period, but it mandates competitions for those jobs as well as provides a fairly narrow definition of who could be hired. Now one has to demonstrate that the scientific and technical needs cannot be met through the regular civil service system-an almost impossible task in principle given the sizable surplus of unemployed or under-employed behavioral and social scientists today.⁽¹²⁵⁾

The effects of this change in hiring criteria can be seen by the elimination of almost all excepted service employees in OERI. In early 1993 there still were approximately 20 excepted service employees in OERI; in June 1997 there were only two excepted service staff members left - mostly due to the restrictions of the new law and the reluctance of the management of OERI to use the new procedures. Interestingly, there has been almost no notice taken by and certainly little protest from educational researchers or policy makers in regard to the elimination of this hitherto innovative way of bringing new ideas and people to OERI on a temporary basis.

One interesting but expensive staffing development is the effort by NCES to create a special outside contractor as a quasi-permanent source of professional and technical help. Many federal agencies already hire outside contractors to perform discrete tasks such as organize conferences, write position papers, and handle travel arrangements for invited speakers. This not only helps the agency to do its work, but it is also a way of overcoming government limitations on staff hiring. Due to substantial overhead expenses, these contracts with outside firms often can be considerably more expensive than if the federal government had been allowed to hire its own employees to do the work.⁽¹²⁶⁾

Seeking to overcome the recent FTE reductions, NCES created a separate entity in 1995 to help with its technical and statistical work-the Education Statistical Services Institute (ESSI). ⁽¹²⁷⁾ ESSI was envisioned as a long-term adjunct of NCES that would work closely with the agency. Approximately 16 of the employees of ESSI in March 1997 had their own offices in NCES - a rather unusual arrangement. Some complaints have been raised that the ESSI personnel are in essence a new type of quasi-government employee and that their presence has diminished opportunities for advancement for regular federal career employees. Questions also have been raised about the relative cost of the operation. Given the overhead for the operation, one rough estimate is that the actual cost of an ESSI employee may be approximately 40 percent higher than if NCES had been able to put that same individual on the federal payroll. Yet at the same time, given the drastic staff reductions in OERI and the increasing volume of business, some new mechanisms need to be developed in order to handle the work of the agency.

Even as OERI lost one-quarter of its staff, there were still considerable opportunities for hiring. It is estimated that in recent years, approximately 7-10 new OERI employees are hired annually.⁽¹²⁸⁾ Some of these are recent college graduates entering federal service through the Outstanding Scholars Program. Staffing from this pool reduces the bureaucratic difficulties of hiring someone (it is much easier to process paperwork for an individual from the Outstanding Scholars Program than to employ someone else from outside the federal government). While these new employees are often quite bright and capable, few of them have had extensive training and experience in research. Other recent staff additions have had prior government service, but most of them also have had little training or previous work experience in research and development. As a result, despite continued staff hiring, few of the new employees would be considered well-trained, nationally recognized researchers and scholars.

As OERI was restructured in 1994 and early 1995, it was necessary to shift at least some of the existing staff to new positions. Sharon Robinson was committed to giving all employees the opportunity to apply for any position in the agency - even though the reorganization of the office was still underway at that time so that management often did not have an adequate sense of the actual staff needs for each unit.⁽¹²⁹⁾ Since approximately 95 percent of the staff were granted their first choice of jobs, some employees ended up in positions where they did not have the necessary experience or technical training to do those jobs.⁽¹³⁰⁾ While all OERI staff were strongly urged to enroll in team building and customer service workshops, they did not receive comparable encouragement or opportunities for taking statistical or technical training that might have been particularly helpful in their new positions.⁽¹³¹⁾

National Research Institutes

The centerpiece of the OERI reauthorization was the creation of the five National Research Institutes. While the proponents of these institutes had envisioned large-scale operations modeled after those in NIH, the actual amount of monies available was rather modest - $43 million for all of them in both FY 1996 and FY 1997 (only about $10 million more than had been spent just on the R&D centers in FY 1995):

Achievement Institute - $12.9 million.

The funds appropriated for three of the five National Research Institutes were well below what many of its original supporters had believed was necessary to set up such institutions. Indeed, some supporters of the National Research Institutes suggested that given the small amount of total funding, only the Achievement Institute and the At-Risk Institute should have been created at this time with a larger budget of $21.5 million each.

The legislation mandated the creation of the five National Research Institutes and provided a long list of possible topics that each institute might research-more than could be reasonably accomplished given the limited monies available. After considerable internal and external discussion, OERI developed its own general, but often not very specific, set of priorities for each institute and established at least one center for each institute.

Initially, the 5 National Research Institutes put most of their $43 million into 7 R&D centers (expanded to 11 centers by FY 1997). While the law stated that they had to spend at least one-third on centers, in FY 1997 they spent $31.15 million on them (57.7 percent). The institutes allocated $14.3 million to field-initiated research - 26.5 percent of the total institute funds. Only $8.55 million was spent on special studies, crosscutting activities, fellowships, or peer reviews (15.8 percent of the total institute budget).⁽¹³³⁾ Thus, the expenditures for the five National Research Institutes were more diversified than that of the former Office of Research (OR), which had focused mainly on R&D centers. Yet even now the R&D centers received almost 60 percent of the entire budget.

There have been significant changes in the size of the annual funding for the R&D centers. In the late 1960s and early 1970s, the R&D centers' budgets averaged between $3 and $5 million annually (in constant 1996 dollars). Over time, average funding for the centers decreased sharply - in the late 1970s and early 1980s averaging between $2 and $3 million annually (in constant 1996 dollars). With the great expansion in the number of centers in the late 1980s, average annual funding for them became even smaller - reaching a low in FY 1991 of about $1 million (in constant 1996 dollars).⁽¹³⁴⁾

The National Academy of Sciences had recommended in 1992 that the annual funding for all R&D centers should be at least $3 million (or $3.35 million in constant 1996 dollars).⁽¹³⁵⁾ Others argued for even more money for R&D centers - perhaps somewhere in the range of $6 to $8 million annually.⁽¹³⁶⁾ Given the general consensus that the R&D centers in the early 1990s were much too small and the specific NAS recommendations for at least $3 million minimum, it was somewhat surprising and quite disappointing that Congress mandated only a $1.5 million annual minimum for these institutions.

While the total funding (in constant 1996 dollars) for the centers remained relatively constant from FY 1992 to FY 1997, the number of them has decreased by more than one half. In FY 1992 there were 22 R&D centers while 5 years later there were only 11 institutions. As a result, the average annual funding (in constant 1996 dollars) of the centers has increased substantially from $1.28 million in FY 1992 to $2.76 million in FY 1997-more than doubling in size.

Looking at the average expenditures is somewhat misleading, as most centers are still very small today. Only 2 of the 11 R&D centers in FY 1997 are well-funded annually: the National Center for Research on the Education of Students Placed at Risk ($5 million) and the National Center for Research on Education, Diversity, and Excellence ($4 million). But 8 other centers are much smaller at $2.5 to $2.8 million each. Moreover, OERI has just funded another R&D center at only $1.5 million.⁽¹³⁷⁾ Given that in the past frequently only half of center funding went directly for research or development expenditures (the rest spent in other areas such as dissemination, administration, and overhead costs), it is likely that serious underfunding is limiting the actual research that can be attempted at these small R&D centers.⁽¹³⁸⁾

It is very important to know how the new R&D centers focus their research and development energies. Unfortunately, there is no in-depth analysis of the number of projects in the new centers or of the actual intellectual cohesiveness among projects within each center. It does appear, however, that many of the new centers include partners at several different colleges and universities - thereby making coordination even more difficult. For example, the new $1.5 million Center on Policy and Teaching Excellence is a consortium including the University of Washington (Seattle), Stanford University, Teachers College at Columbia University as well as faculty at the Universities of Michigan and Pennsylvania.⁽¹³⁹⁾

Based on the original proposal for the National Center for Research on the Education of Students Placed at Risk as well as discussions with several OERI staff members familiar with some of the other centers, one might conclude many of the new centers still seem to support too many projects and projects which are too small; and some of these smaller projects which could be closely integrated with each other are not. Moreover, with a few notable exceptions, there is little evidence that most of the new centers are engaged in the systematic, long-term development of educational materials and models-a continued weakness of OERI in general. Instead, the new centers continue the previous practice of focusing almost all of their attention on pursuing basic and applied research or funding dissemination activities.

While annual funding for many of the individual projects within the centers continues to be only $30,000 to $50,000 annually, the size of the field-initiated grants has grown considerably. The Postsecondary Institute, for example, funded six field-initiated studies in FY 1996 that averaged $210,000 annually (the grants covered 2 to 3 years of work). Thus, whereas one of the more attractive features of the center system in principle was that it concentrated more research monies on a few large-scale, long-term projects that might be done through grants to individual scholars, today the situation may be just reversed-many of the field-initiated grants are larger than some of the particular projects within the centers.

The research plans for many of the individual projects in the old R&D center applications were not well developed or clearly articulated. Some of these projects frequently devoted the first year to a review of the secondary literature; afterwards a more careful, detailed design of the subsequent research work was developed.⁽¹⁴⁰⁾ Moreover, center awards often included projects of very mixed quality - some parts excellent and some parts so weak that they probably would not have survived by themselves a rigorous outside peer review.

Successful field-initiated grants, on the other hand, may have more focused and developed research proposals in order to be able to win the more intense competition for individual funding. And as applicants are expected to be aware of the secondary work in the field, there may be less need to devote the first year of the project to reviewing the existing literature before proceeding to their original work. In addition, as most field-initiated grants are for only 2 - 3 years, they cannot afford the luxury of simply reviewing the secondary literature for the first year before undertaking new research.

Since the center grants focused on a particular problem area, however, it was easier for OERI to target its resources to these sites rather than hold a relatively general and open-ended field-initiated grant competition. The centers also provided opportunities for different researchers to work together on a common set of issues and required them to disseminate their findings more broadly than most individual researchers are willing to do.

These obvious and real advantages to the center approach might be duplicated at least in part by using mission-oriented competitions for field-initiated research. That is, OERI might call for research on a particular area, such as summer learning or the education of pregnant adolescents, and then allow researchers to propose their own topics. Following the practice of many other federal research agencies, the institutes could convene the funded individual researchers to exchange ideas amongst themselves. Moreover, the institutes might also synthesize the results from these mission-oriented individual grants and disseminate the results more widely among other researchers and educational practitioners. In order to administer and intellectually oversee some of these mission-oriented competitions, OERI might hire a leading expert in that substantive area as an excepted-service employee for 3-6 years. In any case, the advantages and disadvantages of funding research through individual grants or the existing R&D center system needs further careful evaluation in the near future.

Given that there are now fewer, but somewhat larger centers, one might imagine that there will be even closer integration of them-especially because OERI was directed to develop a coherent, long-term research priority plan to guide all of their investments. Unfortunately, the results in terms of the integration of the centers have been rather disappointing so far. Faced with the slow development of an overall OERI research priority plan, which will be discussed later, the center competition was held well before any coherent research plan had been developed. And rather than working closely together to create an integrated strategy for a new set of centers, each of the five institutes pursued their agendas and interests quite independently of each other. Moreover, the announcement of the competition for the seven centers in the Federal Register lacked an overall framework or any coherence, so that the applicants and the reviewers for each of the centers treated them in almost total isolation from each other. The net result is that while the 11 centers may be good individually, they have not been integrated conceptually or operationally with each other.⁽¹⁴¹⁾

One factor which has contributed to the lack of coordination among the five institutes (as well as among the other units within OERI) is that no one is really in charge, organizationally or intellectually, of the overall research agenda. Under the old system, all of the R&D centers were located within the Office of Research, which was headed by Acting Director Joseph Conaty, an experienced and respected researcher. Now each of the institute directors reports directly to the Assistant Secretary who is able to supervise them only minimally; no one is responsible for overseeing intellectually the research operations of the institutes as a whole. While various ad hoc schemes have been tried to improve coordination among the institutes (with occasional success), in practice the system has become even more fragmented. Moreover, the expected close coordination between the work of the institutes and the rest of OERI has not materialized except in a few isolated instances.

Adding to the confusion and lack of research coordination among the five institutes is the fact that OERI itself seems to have less intellectual leadership and direction than 5 years ago. Diane Ravitch had been very interested and involved in the intellectual direction and research agenda of OERI; but Sharon Robinson was less concerned and focused more on translating existing research findings for the agency's customers. And whereas Vinovskis helped to coordinate OERI research activities as the Research Advisor to the Assistant Secretary, when he left in August 1993, Robinson did not appoint anyone to replace him. Although in principle Robinson endorsed the concept of an OERI Research Advisor on several occasions, that position remained vacant.⁽¹⁴²⁾ As a result, during the past 4 years there has been a major vacuum in the overall guidance of intellectual and research activities in OERI-one that has become all the more glaring because of the elimination of the Office of Research and the coordinating and leadership role that some of its directors in the past had played.

In terms of staffing, the five institutes have had mixed experiences. The five acting institute directors were selected from the existing OERI staff and most of them had previous research experience - though few were still personally active as researchers. It was expected that OERI would be able to recruit some outside researchers as permanent directors, but the final competition resulted in the appointment of only one outsider as an OERI institute director.

While the overall OERI staff had been reduced by approximately one-fourth from FY 1992, the staffs of the five institutes in FY 1995 were approximately the same size as they had been in the Office of Research. Over time, however, the number of staff members in these institutes has decreased considerably. In June 1997 there were only 64 institute staff members-a substantial decrease from the 77 staff members present in October 1994 (about a one-sixth reduction).

The five institutes fared relatively well within OERI in terms of having generally competent leadership and a sizable staff, but they still lack an adequate number of trained and distinguished researchers. While some current staff have had graduate training and experience in research, most have not. Nor are there many nationally recognized scholars in the institutes. In addition the distribution of the researchers among the institutes is uneven. For example, until very recently the Early Childhood Institute for several years did not have any experienced researchers on its staff even though its mission called for working closely with the research community. Moreover, several of the other institutes had only two or three researchers - hardly enough for a critical intellectual mass to carry out the research and development functions expected of the institutes. As a result, without the national intellectual leadership originally envisioned for the staff of the institutes the production of high quality research and synthesis has not progressed satisfactorily.

One of the primary objectives for the new institutes was to produce high quality research as well as critical syntheses of secondary works that would be helpful to policy makers and educators. So far, most of the institutes have produced only a few publications - usually directories of researchers and service providers, materials for parents, or limited analyses of isolated issues.⁽¹⁴³⁾ Regrettably, the shortage of active researchers in the institutes as well as the current general lack of OERI encouragement of staff to write for scholarly publications means that it may be unrealistic to expect many broad-based analytical studies or syntheses from the institutes in the near future.⁽¹⁴⁴⁾

Despite the fact that the five National Research Institutes were to be the centerpiece of the new OERI, the agency seems to have relegated them to a secondary role. Despite the modest $43 million appropriations for the institutes for FY 1996, the following year OERI did not ask for any increase in that budget-preferring instead to request an additional $250 million to support a new Technology Literacy Challenge Fund which would be used to integrate computer technology into the classroom.⁽¹⁴⁵⁾ Congress, however, added $16 million for OERI in FY 1997. After considerable debate over how that additional money should be spent, OERI allocated $11 million of those funds for the institutes - most of which has gone for an additional center ($2.0 million), more field-initiated studies ($5.5 million), or special studies ($2.8 million).⁽¹⁴⁶⁾

As large programs are added to OERI, the agency has often decided to set them apart from the institutes rather than having the institutes supervise the new activities. For example, although the Achievement Institute is concerned with improving the use of technology in the classroom and specializes in developing better tests, OERI decided not to have that Institute oversee the new high priority initiatives in educational technology or testing. Indeed, staff from the institutes are sometimes temporarily reassigned to work on these new projects under the supervision of the newer unit - thus further diminishing the importance and functioning of the National Research Institutes. As a result, one might easily make the argument that in some ways the five institutes now are less important and less central to the mission of OERI than the Office of Research was 5 years ago - a startling reversal of what the Congress and others had intended for the new OERI. Perhaps even more surprising is the fact that the diminishing of the relative role of the institutes within OERI has gone largely unnoticed and uncontested by the educational research community and interested policy makers.

Office of Reform Assistance and Dissemination

Throughout the long history of NIE and OERI, legislators and educators frequently criticized each agency for failing to disseminate research information to classroom teachers. These critics have often exaggerated the availability of reliable and useful research information about educational practices and have seriously underestimated NIE's and OERI's efforts to distribute that knowledge. Indeed, one might argue that NIE and OERI have tried harder and perhaps have been even more successful than many other federal research agencies in disseminating research information to the public and practitioners. ⁽¹⁴⁷⁾

Prior to the reorganization of OERI in 1994, OERI had been active in improving program development and disseminating the results through its Programs for the Improvement of Practice (PIP). The reauthorization of OERI created the Office of Reform Assistance and Dissemination, which took over most of the functions of PIP. ORAD has four divisions: the state and local Support Division (SLSD), the Knowledge Applications Division (KAD), the Development and Demonstration Programs Division (DDPD), and the Learning Technologies Division (LTD).

Despite the repeated emphasis by policy makers and educators on the importance for ORAD to provide links between research and practice, staffing problems have hampered its operation. Whereas its predecessor, PIP, had 73 employees in 1990, ORAD in March 1997 had only 61 staff members (a one-sixth decrease). Thus, while the Congress and the Clinton Administration envisioned an expanded and improved level of dissemination and reform assistance through ORAD, the restructured operation had substantially fewer employees and lost some of its most effective and experienced managers through early retirement during the buyout period. Even more unsettling and troublesome was the fact that ORAD operated for more than 3 years without a permanent director. Fortunately, Peirce A. Hammond has now been hired as the Director of ORAD.

One of the most important responsibilities of ORAD was to guide and monitor the activities of the regional educational laboratories. There had been considerable controversy surrounding the regional educational laboratories in the past; but some observers were now hopeful that with a new administration a closer and more harmonious relationship might develop between OERI and the labs.⁽¹⁴⁸⁾

Congress reauthorized the regional educational laboratories along similar lines as in the past, but did provide for the possibility of expanding the number of labs from 10 to 12 (which has not occurred). Following the conceptual scheme also employed in other federal educational legislation in 1994, each laboratory was expected to "promote the implementation of broad-based systemic school improvement strategies." Permissible tasks for the labs were enumerated in a lengthy list - ranging from the development and dissemination of educational research products to providing technical assistance and training for state and local educators. Congress again stated ambiguously who should set the research and development priorities of the labs. On the one hand, the labs were to be governed by their own regional boards. On the other hand, they were to respond to the needs of the federal government as set forth in the 5-year lab contacts and to coordinate their activities with other units within OERI such as the National Research Institutes and the National Diffusion Network (NDN).⁽¹⁴⁹⁾

Work on the request for proposals (RFP) for the regional educational laboratories was underway in the second half of 1994 and continued in earnest until it was issued with an August 8, 1995 deadline. Rather than repeating the lengthy list of permissible lab activities, the new RFP called for these institutions to focus most of their resources on supporting broad-based systemic reform efforts, while simultaneously developing a specialized area of expertise. ⁽¹⁵⁰⁾

There were two new goals for the laboratories: (1) to bring together scattered successful reform efforts at the state and local level; and (2) to scale-up the existing successful reforms so that they are adapted by other areas. By the end of the 5-year contract, the laboratories were to have achieved some fairly tangible and ambitious goals. ⁽¹⁵¹⁾

While harnessing the work of the regional educational laboratories to the Clinton Administration's emphasis on systemic reform was understandable, the conceptual basis for that initiative was problematic from the very beginning. The RFP called for each of the 10 regional laboratories to develop, test, and help implement systemic reforms. It would have been more efficient and effective to have one overall coordinated lab effort to develop and test systemic reform models and processes. Otherwise there was likely to be considerable waste and duplication if several different labs assessed identical reform models such as "Success for All," or developed nearly identical educational improvements without using similar measures or procedures to facilitate more scientifically reliable comparisons.

Nor was there a coherent or clear definition of what "systemic" or "comprehensive" reform meant in the RFP. For example, was "opportunity to learn" an integral and essential component of systemic reform? Was the provision of social services to school children also an essential part of systemic reform? Moreover, there were internal inconsistencies even within the RFP on what these terms meant or how they should have been interpreted in practice. The previous labs had interpreted the concept of systemic reform quite differently amongst themselves and some had even explicitly rejected the idea altogether. While the RFP correctly permitted and encouraged different approaches, this might not necessarily lead to meaningful or useful comparisons at the end of 5 years. If OERI wanted to test or develop several different approaches to systemic reform, this should have been done more explicitly and rigorously, so that the relative merits of using different approaches could be better assessed and compared. Without some overall vision or process for developing a more coherent, planned variation approach (including using some standardized measurement schemes), a considerable amount of money and effort might be wasted or misdirected.

Similarly, what was "scaling-up"? Would each of the labs devise their own definition and strategy? Was OERI especially interested in any particular aspect of scaling-up such as the relative costs and benefits of working in partnership with teachers? Would there be any planned comparisons among these different strategies and approaches? Did we really need 10 different, rather uncoordinated, but expensive efforts to study "scaling-up" - especially from an applied research and development perspective?

Perhaps it might have made more sense analytically to subdivide the core lab activities into two broad categories: an applied research and development initiative focused on investigating and assessing systemic reform and scaling-up; and a set of services provided to states and local areas to help them implement systemic reform and scaling-up.

Rather than leaving it up to each lab individually to develop and implement its own, separate applied research and development projects on systemic reform and scaling-up, perhaps they should have been required to work with OERI and through the existing lab network to create a more coherent and coordinated effort. Moreover, this would have encouraged closer cooperation between the labs and the Planning and Evaluation Service (PES) in the Department, who were also working on systemic reform and scaling-up. This way OERI would have minimized unnecessary duplications and ensured that major gaps would be identified and filled. Initially the labs would have made their own suggestions and recommendations on how they would like to proceed, but their proposals then would have been reviewed, negotiated, and coordinated.

The second part of the lab core activities-to assist state and local areas to implement systemic reform and scaling-up-could have been handled in a more traditional manner. Each lab would work within its own region to provide the necessary services for its customers at the state and local levels. While some interregional coordination among the labs still might be useful, it is not as essential for this activity as it is when trying to develop new models and processes which are going to be applicable in different contexts throughout the entire nation. ⁽¹⁵²⁾

The idea of a more organized and coordinated approach to systemic reform and scaling-up was considered at OERI; some of the staff agreed with the conceptual and practical advantages of such an approach. But given the continued hostility of the labs, who preferred to take directions from their own governing boards and lab management, to following the leadership and guidance of OERI, it was unlikely that this alternate approach would have been adopted - even though it might have produced more reliable and useful results in the long run. The labs and their allies in Congress insisted that even though these institutions operated under government contracts and received substantial federal funds, they should remain relatively autonomous and almost totally independent of any specific directives from OERI.

The RFP included several other improvements. It mandated periodic evaluations of the labs - including a much needed independent 3rd-year review of all labs. ⁽¹⁵³⁾ The required quality assurance system for each of the labs was also expanded and made more rigorous. ⁽¹⁵⁴⁾ Another innovative idea was to set aside approximately $200,000 per year for each lab for an optional task 6 - providing assistance to OERI. This was intended as a way to allow OERI to provide more direct guidance for a small portion of the labs' activities and to compensate those institutions when they performed special tasks for the agency. ⁽¹⁵⁵⁾ Finally, the RFP called for each of the labs to create a specialty area development (task 7) - reflecting their own interests as well as those of OERI. Task 7 was important and accounted for one-fifth of the total points assigned by reviewers in the lab RFP. The idea behind this approach was to encourage each lab to develop an area of expertise that would be shared with others across the nation. ⁽¹⁵⁶⁾

While there were areas in which the lab RFP might have been improved, given the political and practical constraints, overall OERI did a good job and made some useful improvements. Certainly the RFP for 1995 was much better than either of the ones for 1985 or 1990. If the specific requirements and recommendations of the RFP were carried out in practice, the orientation and effectiveness of those institutions would have been improved.

When the NAS Panel analyzed the labs in 1992, it stated that the "committee questions the advisability of competitions for RALs [Regional Assistance Laboratories]. The laboratories competed in 1985 and 1990, and only one of the 19 incumbents was unseated in those 2 rounds of competitive bidding."⁽¹⁵⁷⁾

In 1995, however, two incumbents were unseated: Research for Better Schools (RBS) was replaced by Mid-Atlantic Laboratory for Student Success (LSS); and the Regional Laboratory for Educational Improvement of the Northeast and Islands (NE/I) lost to Northeast and Islands Laboratory at Brown University (LAB). In addition, Southeastern Regional Vision for Education (SERVE) was created in 1990. Thus, contrary to the NAS Panel's prediction that there would be almost no changes in the labs through the competitions, there has been a 30 percent turnover over the last 2 competitions. Moreover, there may be a shift in the orientation of some of the labs as the two additions in 1995 are associated with universities.

Not only have three new labs been created, but many of the long-time leaders of the continuing institutions have left. Experienced and influential executive lab directors such as Dean Nafziger (FWL), C.L. Hutchins (McREL), Robert Rath (NWREL), John Hopkins (RBS), and Preston Kronkosky (SEDL) have either retired or been replaced. Indeed, only three of the current executive lab directors were running their institutions before 1990. As a result, the leaders of the regional educational laboratories today are less experienced, but also perhaps more open to instituting changes in the operation of their organizations and reconsidering their relationship to OERI.

Another recent change likely to have a major impact is the resignation of the Executive Director of CEDaR, Dena Stoner, to start her own trade association. While Stoner hopes to continue to work with some of the CEDaR center and lab members in her new post, it is not yet clear what will happen. Moreover, the Department is reexamining and questioning the legality of the previous financial relationship between CEDaR and the labs and centers; this may also lead to a restructuring of that connection in the future. How these recent changes in CEDaR will affect the operation of the labs as a whole or the political influence which the labs have had with the 105th Congress and the Clinton Administration remains to be seen.

At the same time that there have been significant changes in lab membership and leadership, there have also been indications that some of their overall goals and practices have remained basically the same. With the Republicans gaining control of Congress after the 1994 elections, some observers anticipated the implementation of more critical oversight of the labs; but lobbyists for the labs have operated almost equally effectively with the new Republican Congress. For example, CEDaR managed to persuade its congressional allies to shift the $10 million saved by eliminating the National Diffusion Network in FY 1996 to the labs. While some have portrayed this change as a temporary transfer, most observers expect that the additional $10 million will become a permanent part of the federal funding of the labs-just as the special monies for the rural initiatives in the late 1980s were incorporated in the regular lab budget over time.

Similarly, despite the apparent support of the labs and CEDaR for the new directions enunciated in the 1995 RFP, once the labs received their new contracts some of them balked at working on projects initiated by OERI (task 6). The OERI staff were caught off-guard by the discovery that House and Senate appropriations committees had been persuaded to repudiate the explicit terms of the 1995 lab contracts. Congress in its appropriations reports insisted that all of the work of the labs should be based only on the priorities established through their own regional governing boards.⁽¹⁵⁸⁾

Besides working with the regional educational laboratories, ORAD was also charged with identifying promising educational models, developing and testing them in different settings, and then disseminating that information. Unfortunately, the Department in general and NIE or OERI in particular, have failed to develop and rigorous test educational models in the past. As Robert Slavin recently put it:

• For decades, policymakers have complained that the federal education research and development enterprise has had too little impact on the practice of education. With few notable exceptions, this perception is, I believe, largely correct. Federally funded educational R&D has done a good job of producing information to inform educational practice, but has created few well-validated programs or practices that have entered widespread use.⁽¹⁵⁹⁾

OERI has done little to support the systematic and rigorous testing of alternative educational models. Instead, it has spent most of its monies on a series of small, short-term projects (including much of those funded through the labs and centers). While OERI in mid-1994 had received and approved a plan for a large-scale, rigorous evaluation of systemic reform, it failed to develop and implement that proposal.⁽¹⁶⁰⁾

As a result, when the Clinton Administration focused on assessing the strengths and weaknesses of systemic reform, it relied more heavily on the work of the Planning and Evaluation Service in the Department, especially its recently funded Longitudinal Evaluation of School Change and Performance. OERI has played a very minor role in these discussions - in part because the agency itself was not really undertaking any large, scientifically rigorous studies of systemic reform. Thus, ironically, whereas OERI might have been one of the leaders in the efforts to critically assess systemic reform in mid-1994, it failed to take advantage of that opportunity and encouraged the Department to look elsewhere for intellectual leadership and assistance on how to assess the impact of systemic reform.

National Center for Education Statistics

During the OERI reauthorization process, the Congress did not devote as much attention to NCES as to other divisions within the proposed new agency such as the National Research Institutes and the Office of Reform Assistance and Dissemination. The National Center for Education Statistics continued to be seen as the primary federal agency for collecting domestic and international educational data.⁽¹⁶¹⁾

The National Center for Education Statistics (NCES) was divided into four units: the Statistical Standards and Methodology Division; the Data Development Division; Education Surveys Division; and the Education Assessment Division. NCES also made effective use of advisory groups - including two key ones which were legislatively mandated: the Advisory Council on Education Statistics (ACES), which helped the Commissioner with general policies and operating standards; and the National Assessment Governing Board (NAGB), which oversaw the National Assessment of Education Progress (NAEP).

During the decade that NCES had made such important and meaningful improvements, it also faced some serious challenges. Perhaps one of foremost was the decision of Emerson Elliott to retire at the end of his term as the first presidentially-appointed Commissioner of Education Statistics. As we discussed earlier, Elliott was one of the most experienced and effective career federal civil service managers and played an instrumental role in developing NCES into a first-rate statistical agency. Pascal (Pat) Forgione, Jr., the former Delaware State School Superintendent, was a good choice to replace him on July 1, 1996.⁽¹⁶²⁾ But Forgione understandably is relatively inexperienced and does not yet have the extensive network of contacts Elliott had built up during his three decades in the federal government. Nor does he have a close working relationship with many of the OERI leaders and middle-level staff that Elliott had developed over the years. On the other hand, Forgione appears to be very knowledgeable and interested in improving the technical aspects of the NCES work.

Another troublesome problem for NCES was the unexpected, dramatic decline in staff size. When the NAS Panel called for a doubling of the size of the NCES staff, the agency had about 143 employees in 1991. By October 1994 NCES had decreased to 112 and by March 1997 it was still only 115 - an overall decrease of nearly one-fifth. Thanks to the open enrollment season in September 1994 as well as the buyout program for senior staff, NCES lost some technically skilled and experienced employees. Recently some of the more competent NCES staff have left the unit to work elsewhere in OERI or in other parts of the federal government - partly in reaction to recent changes at NCES.

The general de-skilling of the rest of the OERI research staff also meant that it is now more difficult to recruit NCES staff from elsewhere in the agency-an important source of recruitment in the past. OERI has temporarily "borrowed" a few of the more accomplished NCES professionals to work on high priority projects such as the new voluntary national education testing initiative.⁽¹⁶³⁾ And while the new Education Statistics Services Institute provided help on some of the more technical aspects of the work, it also created additional problems and increased staff tensions that have yet to be resolved. Thus, Commissioner Forgione was faced with having to satisfy the increased mandated tasks for NCES with a staff that recently has been considerably diminished and weakened.

As the expectations and responsibilities for NCES expanded substantially in recent years, increased funding for those activities has not been adequately provided. NCES funding rose dramatically from $19.5 million in FY 1987 to $85.5 million in FY 1993 (in constant 1996 dollars). But since then funding has actually decreased to $80.6 million in FY 1997 (in constant 1996 dollars) - a 5.7 percent decrease. As a result, NCES's more ambitious data collection and dissemination agenda has had to be implemented by a smaller staff with fewer real dollars.

Despite these new challenges, NCES has managed to increase the diversity and usefulness of its publications for multiple audiences while producing a high quality work. It is interesting to observe that when Assistant Secretary Robinson testified before Congress, most of the OERI publications she distributed to the members were from NCES rather than from the National Research Institutes or ORAD. At a time when the other units within OERI are having great difficulty in doing either original research or critical syntheses of existing scholarship, NCES continues to make significant contributions not only to the collection of data, but to its analysis as well.

NCES also has made some key improvements redesigning the National Assessment of Educational Progress. Congress mandated NAEP in 1969 and national tests at grades 4, 8, and 12 have been administered for 10 different subjects. In 1990 some NAEP tests were also administered and released at the state-level and this component has expanded rapidly. In 1996, 44 states voluntarily participated in the NAEP math assessment at grades 4 and 8 and in the science assessment at grade 8.⁽¹⁶⁴⁾ Moreover, the National Assessment Governing Board (NAGB) has called for even more linkage of the NAEP tests to state and local needs.⁽¹⁶⁵⁾

Following up on this suggestion, NAGB announced, in March 1997, a tentative national and state schedule for the NAEP tests until the year 2010. The national tests in the 10 subjects would continue periodically, but they would be supplemented by state tests in reading and writing or mathematics and science every 2 years.⁽¹⁶⁶⁾ In order to expand the number and type of tests and yet stay within the existing budget, NCES is considering alternative ways of redesigning and administering the tests in order to save money.⁽¹⁶⁷⁾

While NCES is doing a good job in providing new and expanded coverage of subjects like reading, writing, mathematics, and science at the state-level, it is neglecting other important areas like social studies. For example, national U.S. history or geography NAEP exams are scheduled only for 2001 and 2009 and the civics exams in 1998 and 2003; the world history exam will only be given only in 2005. Moreover, none of the social studies subjects are now scheduled to be included in any of the NAEP state assessments.⁽¹⁶⁸⁾

The neglect of social studies is not unique to NCES. The Department of Education in its plans for national tests as well as its evaluation of the Title I program, has focused almost exclusively on reading and mathematics assessments. While the Congress and the Clinton Administration frequently have evoked the importance of improved K-12 education for good citizenship, in fact they have not provided much leadership or guidance in this area. Most of the public and policymakers appear to be unaware of this rather narrow definition of the subjects to be assessed; nor have professional social studies organizations protested the relative neglect of social studies.⁽¹⁶⁹⁾

NAEP could also be improved by providing better information on the economic status of the students being tested. For example, the 1994 U.S. history scores were reported by the race and ethnicity of the students, but not by the income of their parents. One-half of white students, 83 percent of black pupils and 78 percent of Hispanic students in the 12th grade performed below the basic level.⁽¹⁷⁰⁾ Yet undoubtedly some of these racial and ethnic differences in knowledge of American history reflected the difficulties low-income students may have had in doing as well as their more fortunate middle-class counterparts.⁽¹⁷¹⁾ From reports based on race and ethnicity, but not on the economic status of the family, policymakers and the public have received a rather limited and misleading impression of the basic factors which help to explain the differential performance of elementary and secondary school students on the national history test.

NAEP has plans to ascertain whether or not a tested student participates in the subsidized food lunch program in order to get some indication of poverty status. Unfortunately, this is not a very reliable index and needs to be supplemented or replaced by better indicators. Naturally, it is difficult to acquire valid measures of parental income from elementary students. Efforts should be made to experiment with alternative ways to retrieve that information (perhaps from a more in-depth questionnaire to a sample of the parents, or from student school records which may have information on parental characteristics). Rough estimates of parental income might be asked of 12th-grade students as they may know the approximate income of their families. In any case, NCES and NAGB need to devote more attention to finding more appropriate indicators of the economic well being of the parents of the students.

Another useful, but potentially difficult new direction at NCES is to expand its work in policy analysis. NCES has always been involved in doing policy analyses, but the agency has been reluctant to expand too far in this area as it feared compromising its independence and objectivity by being too closely identified with a controversial policy or a particular administration.

As NCES expands the use of NAEP data to the state-level and works more closely with state and local officials, support for that agency probably will grow as the usefulness of its work becomes more apparent. At the same time, however, there are hidden dangers in this new approach. Political leaders may express dissatisfaction with the state NAEP results if they should happen to contradict the often more optimistic findings from a state's own educational assessments. Policy makers also want results from the NAEP tests to be processed more quickly and presented in less technical language - putting more pressure on NCES to work faster and create additional, less complicated reports. And working more closely with state and local educational policy makers may necessitate providing those officials with additional training and assistance. None of these problems is insurmountable, but each will require additional NCES staff and resources, which are increasingly in short supply at the agency. Finally, will doing these more expensive, policy-oriented investigations drain scarce resources from the other more basic and traditional NCES data collections and statistical studies?

As NCES undertakes more policy analyses, it will also necessitate using more complex statistical techniques to ascertain the relative impact of different factors. While NCES has done a relatively good job of collecting data and accurately reporting them, the agency has been quite backward in using more sophisticated statistical methods of analysis. Most NCES studies still rely almost exclusively on a simple cross-tabulation of the data. Only recently have a few of the NCES analyses even employed multivariate statistics and more complex modeling techniques. ⁽¹⁷²⁾ And few NCES studies do an adequate job of reviewing and critically interacting with the secondary literature on the topics they are addressing.

As NCES moves toward more sophisticated and complex policy analyses, it will face many additional challenges. Will NCES be able to hire more staff with the needed substantive knowledge? Right now its technical research staff can go from one project to another rather easily because they are not necessarily expected to be experts in any particular subject area. But requiring staff to have substantive as well as statistical expertise not only increases the costs of hiring such individuals, but limits the number of different areas which anyone can be reasonably expected to cover. Similarly, will NCES have to create additional advisory panels which have a more substantive orientation in order to help oversee this new work? While ideally NCES would work closely with the OERI National Research Institutes to obtain some of that substantive expertise, unfortunately the shortage of first-rate researchers in those units reduces the likelihood of receiving too much help from these organizations. Thus, as NCES moves more into substantive and policy analyses, its staffing and organizational structure will need to be reassessed and perhaps more coordinated with related developments in other OERI programs than in the past.

Finally, as NCES moves more into policy-oriented work, it will face criticisms from those who may disagree with the particular initiatives. For example, the Clinton Administration has been pushing actively for developing voluntary national tests for reading and mathematics. While this initiative has attracted considerable political support, it has also raised significant opposition. OERI has been assigned the task of overseeing the development of these tests and has relied heavily for assistance upon some prominent NCES staff members. Will critics of the tests someday attack NCES for becoming too involved in specific Administration policies? Will these same critics try to withhold additional discretionary funding from NCES for fear that these monies will be spent on projects which they deem are too partisan or one-sided?

The National Educational Research Priorities Board and the OERI Research Plan

As we have seen, protracted controversy over the characteristics and power of the National Educational Research Policy and Priorities Board (NERPPB) delayed the reauthorization of OERI by almost 2 years. Yet the NERPPB was seen as so important by some members of Congress that they felt it was worth the delay to ensure that a powerful and active OERI policy board was created. The final compromise created a strong advisory board, but the actual role the board would play in the agency depended to a large degree on who was to be appointed to that group and how they would relate to OERI in practice.

Given the centrality of the policy board in the legislation as well as the planned major restructuring of OERI, it was disappointing that the Department moved so slowly to appoint the Board. The legislation mandated that the Board had to meet by May 15, 1995, but Assistant Secretary Robinson had hoped to have it appointed by November 1994.⁽¹⁷³⁾ Finally, on February 27, 1995 Secretary of Education Richard Riley named 15 members from over 300 nominees to the new National Educational Research Policy and Priorities Board. As mandated by Congress, five of the appointees had extensive experiences in elementary and secondary education. ⁽¹⁷⁴⁾ Another five were selected for their broad educational expertise and background.⁽¹⁷⁵⁾ And five were education researchers who had been nominated by the National Academy of Sciences.⁽¹⁷⁶⁾ Many of the key educational lobbyists were quite enthusiastic about the quality of the appointments to NERPPB.⁽¹⁷⁷⁾

The Board held their first meeting on March 30-31, 1995. Members of the Board worked well together and demonstrated a healthy respect for the varied goals of OERI and their own diverse backgrounds. Kenji Hakuta, representing the research community, and John T. Sanders, reflecting the broader educational orientation of the Board, were unanimously elected co-chairs. The Board also created four ad hoc subcommittees: (1) Research Priorities, (2) Regional Educational Laboratories, (3) Standards, and (4) Research and Development Centers.⁽¹⁷⁸⁾

Some critics of the NERPPB had been concerned about the quality of the interactions between the Board and OERI. But thanks in large part to the openness and personal effectiveness of Assistant Secretary Robinson as well as the full co-operation of the members of the Board, a close and harmonious working relationship quickly developed between the two groups. John Christensen, a high-level, senior OERI professional, initially was designated as the federal liaison to the board and played a key role in facilitating NERPPB activities. Eve Bither, a former high school physics teacher, school administrator, and state school superintendent as well as the acting Director of ORAD, was selected by the Board to be their Executive Director in June 1996. ⁽¹⁷⁹⁾ Bither's extensive experiences in state and federal educational affairs as well as her ability to work effectively with diverse interest groups has helped to maintain a close and cordial working relationship between OERI and the Board.

Perhaps the primary responsibility for NERPPB was to work with OERI to develop a 5-year research priority plan for the agency. Unfortunately, much of the anticipated expenditures for OERI for the next 5 years had already been planned and almost fully committed before the Board was even appointed. For example, although several of the Board members at their first meeting raised questions about the wisdom of allocating so much of OERI's funds to the labs and centers, they did not have adequate time to investigate and discuss these issues as the announcements for the competitions for these two sets of institutions were due in the next couple of months.⁽¹⁸⁰⁾ Board members also were disappointed to learn that OERI had not yet developed a preliminary long-range research plan even though much of the agency's monies for the next 5 years was already being committed. The lack of an overall OERI research plan was surprising because the agency had been periodically working on one for the past 3 years.

The legislation mandated that by October 1, 1995, OERI and NERPPB publish a report specifying the agency's research priorities for the next 5 years. Yet as we have seen, NERPPB met for the first time in March 1995 and only then began to think about a research priority plan. Under these circumstances OERI and NERPPB decided to publish a general research framework as an interim product by October and then use the following year to provide a more specific and detailed future research proposals.

The newly created OERI Research Priorities Planning Team presented NERPPB with a set of alternatives and recommendations on a series of important issues in June 1995. One important question the OERI Planning Team addressed was whether there should be an overarching framework or perspective for the plan.⁽¹⁸¹⁾ The Research Priorities Planning Team recommended, and NERPPB in essence accepted, the idea that there would be no overall framework for the 5-year research plan. Instead, OERI developed future research plans which were relatively separate for each of the agency's units.

The decision not to try to develop an overall framework was unfortunate because it continued to leave OERI's operations fragmented and uncoordinated at a time when many policymakers had hoped to focus the agency's resources on a few, major initiatives. An alternative approach had been proposed and almost accepted by OERI in early 1995. In his OERI-commissioned essay, Vinovskis developed a life-course perspective that elicited considerable interest within the agency. The proposed life-course framework offered a more dynamic and yet comprehensive approach and emphasized focusing reform efforts on the more crucial transitions in the lives of children.⁽¹⁸²⁾

The life-course framework was discussed widely within OERI and endorsed by many of the staff. No alternative comprehensive frameworks emerged during those meetings to challenge this approach. And aspects of the life-course approach were even adopted by some units in their own planning. But ultimately the lack of overall intellectual leadership at OERI meant that most programs continued to pursue their own interests and orientation without trying to fit their work within any larger overall context.

After 18 months of drafting and crafting, the 5-year research priorities plan for education research was released jointly by OERI and the National Educational Research Policy and Priorities Board (NERPPB) in December 1996. The 112-page plan, Building Knowledge for a Nation of Learners, was colorfully illustrated, beautifully published, and widely distributed. The report listed seven national priorities for research in education:

1. Improving learning and development in early childhood so that children can enter kindergarten prepared to learn and succeed in elementary and secondary schools.

2. Improving curriculum, instruction, assessment, and student learning at all levels of education to promote high academic achievement, problem-solving abilities, creativity, and the motivation for further learning.

3. Ensuring effective teaching by expanding the supply of potential teachers, improving teacher preparation, and promoting career-long professional development at all levels of education.

4. Strengthening schools, particularly middle and high schools, as institutions capable of engaging young people as active and responsible learners.

5. Supporting schools to effectively prepare diverse populations to meet high standards for knowledge, skills, and productivity, and to participate fully in American economic, cultural, social, and civic life.

6. Promoting learning in informal and formal settings, and building the connections that cause out-of-school experiences to contribute to in-school achievement.

7. Understanding the changing requirements for adult competence in civic, work, and social contexts and how these requirements affect learning and the futures of individuals in the nation.⁽¹⁸³⁾

While the seven national education research priorities are well stated and at times perhaps even inspirational, they are often so general as to offer little specific guidance on where scarce research funds and energies should be focused. For example, no one could disagree with the second priority, which calls for "improving curriculum, instruction, assessment, and student learning at all levels of education." But this declaration does not provide any suggestions for which subjects or at what levels of education efforts should be focused to help children learn better. Nor does the report summarize effectively what is known about this entire area from a research perspective or what specific research gaps should be addressed in the next 5 or 10 years. Moreover, the report acknowledged that "[i]n setting this agenda, the Assistant Secretary and the Board refrained from ranking the priorities."⁽¹⁸⁴⁾

When the priorities do provide some general guidance and direction, as in the emphasis on early childhood education (priority number 1) and middle and high schools (priority number 4), it is not clear why these three levels of education were selected for special attention while others were ignored. For example, many scholars and studies have suggested that K - 3 education is a particularly important period of education for helping disadvantaged children. Indeed, OERI's continued assistance in the development of "Success for All" has pointed out the importance of these grades in helping at-risk children.⁽¹⁸⁵⁾ Yet the National Priorities for Research in Education seemingly slighted K-3 education without explaining to the reader why. Was it because they thought that improving prekindergarten education and reforming middle and high schools had a larger impact on students than helping K-3 education? Or did they think we already know enough about K-3 education so that we do not need any more research in this area? And if K-3 education is not one of the highest priorities for OERI, why does the agency continue to fund "Success for All"?

Research priority number 6 focuses appropriately on the contribution of out-of-school learning experiences for the development of the child. Certainly this is one of the more crucial areas for further research as studies have pointed to the importance of learning outside the classroom. But in the report's discussion of this topic, the focus is mainly on extracurricular activities, after-school opportunities, and parental involvement in school programs. ⁽¹⁸⁶⁾ No mention is made of the potential importance of summer learning and the problems that at-risk children have during this period - even though the existing research suggests that this might be one of the most important factors in explaining the relative disadvantage of learning for children from low-income families.⁽¹⁸⁷⁾ Thus, a reader might wonder just how carefully and comprehensively the existing research was consulted and incorporated in making these specific recommendations.

One also wonders about the general tone of the report. Sometimes it appeared to be too complacent about the quality and quantity of existing research.⁽¹⁸⁸⁾ And while the report stressed the need for improvements in the future, its defense of school achievements in the past quarter of a century seems somewhat strained and exaggerated.⁽¹⁸⁹⁾ Moreover, while the report generally did a good job of emphasizing the problems that many minority students face today, surprisingly it often seemed to ignore or downplay the negative impact of poverty on all children.⁽¹⁹⁰⁾

Overall, while the National Priorities for Research in Education was clearly written and handsomely produced, it does not yet provide a very useful set of guidelines or suggestions for educational research for the next 5- or 10-years. Many of the OERI staff privately acknowledge that Building Knowledge for a Nation of Learners has not had much of an impact even within the agency. In a recent OERI Bulletin, for example, Ramon Cortines, then acting Assistant Secretary, listed seven priorities of the Department without even mentioning the existence of the new OERI research priorities.⁽¹⁹¹⁾ And there was little evidence that those in other units of the Department or outside the federal government have paid close attention to its recommendations. While some of the responsibility for the shortcomings of the OERI research plan must rest with NERPPB, most of it belongs to OERI, which has had so much difficulty in developing and implementing any sustained, in-depth intellectual agenda.

Development of Quality Standards for OERI Work

One of the more interesting and useful provisions of the legislation reauthorizing OERI in 1994 was the requirement to establish high standards of professional excellence for its research, development, and dissemination products and activities. The close cooperation of OERI and NERPPB in drafting these regulations also reveals how both organizations have made a very important, but as yet little noticed contribution to improving the quality of the research, development, and statistics sponsored and overseen by the federal government.⁽¹⁹²⁾

The legislation specified three phases for the development and implementation of the standards for the conduct and the evaluation of research. The first phase was to be completed within the first year and specified that the process by which grants, contracts, and cooperative agreements should be peer reviewed and awarded.⁽¹⁹³⁾ By the end of the second year, standards were to be drawn up for the exemplary and promising practice programs.⁽¹⁹⁴⁾ Finally, by the conclusion of the third year, the Assistant Secretary would develop standards to periodically evaluate the performance of all OERI-funded activities.⁽¹⁹⁵⁾

OERI has almost completed the first two phases of creating standards and overall the agency has done a good job. The phase one standards are related to those in phase three. The phase one standards are broad and cover all new grants, cooperative agreements, and contracts; they are the beginning point for all projects eventually funded by OERI.⁽¹⁹⁶⁾ While the proposed phase two standards are thoughtful and quite important, they focus almost exclusively on the special exemplary and promising practice programs.⁽¹⁹⁷⁾

The phase one standards provide the procedures and guidelines for the evaluation of applications for grants, cooperative agreements, and proposals for contracts. As intended by the Congress, the activities covered by the standards are broad and seemingly all-inclusive. One major ambiguity, which is not specifically addressed in the document, is whether the work of the National Center for Education Statistics (NCES) is covered by these standards. On the one hand, it does not appear that the Congress explicitly exempted the work of NCES from these standards. On the other hand, NCES frequently has its own regulations-often more scientifically rigorous than those employed in some of the other OERI units.

Assistant Secretary Robinson decided as a policy matter that the phase one standards should be applied to all OERI programs. Secretary of Education Riley then extended the phase one standards to all of the Department's activities. The policy decision to apply these standards broadly was commendable and seems to have been useful in practice.

Implementing phase one and two of the standards was not overly onerous because the tasks were more easily defined and logistically manageable. For example, rather than having to list all of the different sets of criteria for every possible type of grant competition, the phase one standards simply provided a menu of criteria to chose from and permitted flexibility in assigning weights to them. Since the criteria and assigned weights for any competition were available ahead of time, both the applicants and the reviewers had a clear idea of what was expected of them. It was also reasonable to expect to find a sufficient number of qualified outside peer reviewers since there were only a few applicants for each of the large-scale grants and the smaller ones were shorter in length and less-detailed (hence requiring less time from reviewers). Moreover, applicants for funding of less than $100,000 were not required to be reviewed by peers.

The situation in regard to the phase three standards is somewhat different and much more daunting in scale and scope. All grants, cooperative agreements, and contracts are to receive both an interim and a final evaluation (not just those over $100,000). Given the great diversity of OERI projects, both in terms of the scale and type of activity, it will be difficult, if not impossible, to provide adequate criteria in the regulations without either resorting to a menu approach or allowing greater specificity by subdividing activities by the type of recipient or the type of product produced. If one decides to depend heavily upon outside peer reviewers, it may be difficult to recruit and compensate sufficient numbers of qualified individuals. As a result, before drafting the final phase three standards, OERI should develop an overall evaluation plan which ensures that all of its research, development, and dissemination funded-activities will be assessed adequately and efficiently.

OERI's overall process for developing the phase three standards also needs some review. The OERI staff and NERPPB generally have done a good job of raising important issues such as the development of an OERI quality assurance system in order to extend the 5-year center cooperative agreements without recompetition. They have also succeeded in soliciting the views of some of the individuals representing the interests of the centers and labs.⁽¹⁹⁸⁾

But surprisingly little, if any, attention has been paid to the difficult problems involved in OERI creating its own system for monitoring and evaluating all of its programs and activities. By separating in practice the drafting of the phase three standards from the development of program oversight procedures and practices, OERI and NERPPB may have inadvertently missed an important opportunity to investigate and resolve the difficult problems inherent in the creation of a viable quality assurance system. Have OERI and the Board analyzed the strengths and weaknesses of the agency's past monitoring and program evaluation practices? Why has the NCES review system seemed to work better than those developed for the labs and centers? Have the OERI and the Board explored the particular quality control problems that are believed to have plagued educational research and have they discussed how this might affect the phase three standards? In addition to hearing from representatives of the major institutions funded by OERI, will the Committee on Standards be hearing from some of the critics who have questioned the quality of much of NIE/OERI's work over the past 25 years? Thus, while OERI staff and NERPPB have made a good beginning in drafting the phase three standards, they also need to address further some of these broader, related questions.

Finally, it is important that, as much as possible, efforts should be made to develop and rely in part upon the quality assurance systems developed by the grantees and contractors themselves (with the guidance, assistance, and oversight of OERI). These self-evaluation systems then can be periodically assessed to ensure their viability and usefulness and thereby greatly reduce the cost and logistical problems involved in trying to create an overall quality assurance system. Otherwise, the cost and reporting burden of the phase three standards may become too high to ensure their full development and implementation. Moreover, as the phase three standards are developed they should also be linked to the collection and synthesizing of information from the OERI-funded recipients. After all, the ultimate goal of trying to develop high quality research, development, and dissemination is to use it substantively and effectively in improving American education today.

Recent Developments at OERI

During the past year there have been some developments at OERI that have raised concerns among its supporters. Sharon Robinson resigned as the Assistant Secretary after 3 - years in office - one of longest terms of any head of NIE or OERI in the past 25 years. After this lengthy period of stability in leadership, OERI seems to have reverted to the troublesome practice of numerous, short-term assistant secretaries. For almost a year after Robinson had left, the White House had not nominated a successor (and her decision to leave was communicated to the Administration even earlier).⁽¹⁹⁹⁾ Instead, OERI has had three acting assistant secretaries - Marshall "Mike" Smith, the Acting Deputy Secretary of the Department of Education; Ray Cortines, the former Superintendent of New York City and San Diego schools and a special consultant to the U.S. Department of Education; and now Ricky Takai, a professional staff member of the Planning Evaluation Service (PES) in the Department. Kent McGuire, a program officer at the PEW Charitable Trusts K-12 education reform and restructuring program, was nominated in late October 1997 to be the next OERI Assistant Secretary.⁽²⁰⁰⁾ Unfortunately, the nomination occurred so late in the legislative session that the Senate did not act on his confirmation until June 1998. Thus, in little under two years, OERI has already had five different assistant secretaries - the greatest number of changes ever in annual leadership in NIE or OERI.

In addition to the lack of continuity due to the ever-changing leadership, there is growing concern that OERI may be losing some of its long-sought independence and distance from policies and directives of any particular administration or Congress. Usually someone from within NIE or OERI has been chosen to serve as the acting assistant secretary as the White House nominates an individual for that permanent position. Yet the Clinton Administration now seemed to many observers have selected acting assistant secretaries who are closely identified with the top political leadership of the U.S. Department of Education. The implicit and troubling message also is that no one at OERI was qualified to serve even temporarily as the acting assistant secretary of the agency.

Concerns about the increasing use of OERI by the Clinton Administration to support its more immediate educational agenda have been fueled by the controversial decision to have the agency develop and oversee the voluntary individual national tests in fourth-grade reading and eighth-grade math using OERI staff and discretionary funds. Many Republican and Democratic members of the House of Representatives have denounced the proposed national tests and are insisting that any such major educational undertaking should be first discussed and then authorized by the Congress before the Administration proceeds further on this matter. Representative Bill Goodling (Republican-Pennsylvania), Chair of the House Committee on Education and the Workforce and an ardent opponent of the tests, stated that "[i]t's probably the most controversial issue to come before Congress this year."⁽²⁰¹⁾ Fortunately, it now appears that the Administration and the Congress have reached at least a temporary compromise on the national testing issue which appears to have removed OERI from the direct development and supervision of that highly controversial undertaking.⁽²⁰²⁾

Another Congressman's question about the national tests revealed an interesting new perspective on the responsibilities of the National Educational Research Policy and Priorities Board. Representative Jay Dickey (Republican-Arkansas), in an appropriations hearing on March 1997, asked Acting OERI Assistant Secretary Smith about the decision to use the discretionary monies from OERI's Fund for the Improvement of Education (FIE) for developing the national tests. Dickey wanted to know if NERPPB had "deliberated on the priority and included it in their plan."⁽²⁰³⁾ Smith acknowledged that it was not part of the OERI research priority plan, but pointed out that the Administration had made a quick decision to go ahead with the proposal before the Board had an opportunity to meet and discuss it. Moreover, Smith went on to indicate that according to his interpretation of NERPPB's responsibilities, the Board did not have jurisdiction in this area:

• ... the Priorities Board is a research board. The Fund for the Improvement of Education is not in the research side of the agency. Think about OERI having three sides, one's a research side, one's a side that does development and dissemination, and then there's the statistics side. It is in the development and dissemination side of the agency.

• And the Fund itself gives broad authority to the Secretary, in effect, to carry out activities, a wide range of activities, to improve the quality of education.⁽²⁰⁴⁾

Upon further questioning, Smith indicated that the Secretary had the power to spend these discretionary funds "without referral to the board," but promised to look further into that matter.⁽²⁰⁵⁾

The debate over the funds for developing the national tests revealed that the leadership of the Department had a rather limited view of the powers of the new Board-apparently seeing its role as mainly to oversee only the research side of the OERI and not its development, dissemination, and statistical functions. This interpretation seems at odds with the general tenor of earlier congressional discussions of the proposed policy board and contrary to the spirit of how Assistant Secretary Robinson had treated the NERPPB. Whatever the actual legal resolution to this important question, it may signal a possible shift in the nature of the future relationship between OERI and the Board as new assistant secretaries are appointed.

Finally, the recent heavy emphasis of OERI management on developing and implementing the voluntary national reading and math tests as well as other new initiatives like the expansion of educational technology has also raised some questions about the overall priorities within the agency. Some of the staff fear that the more traditional research, development, and statistical activities in the National Research Institutes, ORAD, and NCES might be further slighted as the attention of the new leaders becomes focused on the more pressing, immediate needs of the Department.

These recent developments involving the lack of stable leadership at OERI, the reduction in the relative independence of the agency from specific administration or congressional policy initiatives, and the overall setting and implementing of OERI's research and development priorities threaten to hurt the agency in the long-run. Clearly there is a need now for a distinguished educator and researcher to be the next assistant secretary who will provide the intellectual leadership necessary to enable the agency to help all children thrive in our schools and society. And steps should be taken to protect the scientific integrity and relative political independence of OERI lest the agency unintentionally and unnecessarily again become viewed as being too politicized.

[VI. Changes During the Bush Administration]

[VIII. Concluding Observations]