A r c h i v e d  I n f o r m a t i o n

The Board's Goals and Recommendations for Educational Research

In the 1994 "Educational Research, Development, Dissemination, and Improvement Act," Congress set forth a powerful challenge for education research and for the National Educational Research Policy and Priorities Board:

The Congress declares it to be the policy of the United States to provide to every individual an equal opportunity to receive an education of high quality... To achieve (that) goal... requires the continued pursuit of knowledge about education through research, development, improvement activities, data collection, synthesis, technical assistance, and information dissemination. While the direction of American education remains primarily the responsibility of State and local governments, the Federal Government has a clear responsibility to provide leadership in the conduct and support of scientific inquiry into the educational process...The failure of the Federal Government to adequately invest in educational research and development has denied the United States a sound foundation of knowledge on which to design school improvements...(The) National Educational Research Policy and Priorities Board should...work collaboratively with the Assistant Secretary to forge a national consensus with respect to a long-term agenda for educational research, development, dissemination, and the activities of the Office.

As a result of its studies, and its meetings with teachers, educators, researchers, policymakers and others, the Board has reached consensus on four goals that are critical to meet the challenges expressed by Congress in the 1994 legislation. These goals are statements about characteristics of research in education. If the goals were reached, there would be a sound basis for trust in the results and growing support for conduct of research. The Board's goals are:

• Priorities are set and activities are problem-centered.

• High standards of quality are created and upheld.

• Work is collaborative and rigorous.

• Mission is congruent with resources.

The sections below provide additional information about each of these goals and make recommendations for action to achieve them.

Priorities Are Set and Activities Are Problem-Centered

In the Board's work with the Assistant Secretary to establish a priorities plan, and its subsequent efforts to refine and target priorities, it has found that a problem-centered focus for research for developing research agendas works best. That is, identifying real problems faced by teachers in real classrooms is the most understandable way to design and target appropriate research. The principal priority should be teaching and learning and, more specifically, improved achievement for all students. But balanced research agenda setting must also give weight to identification of research opportunities—where research is poised for advances. The targets for action should be ones for which there is reason to be optimistic that research has something important to say, or could have, with the proper investment.

Together, the Board and OERI sponsored a 1998 conference on "National Directions in Education Research Planning" that brought together leaders and representatives from a dozen or more research-planning efforts under way among federal agencies, professional and scientific organizations. Its purpose was to put individuals associated with those efforts into communication with one another and with the educators and policymakers who could use the fruits of education research to enhance learning and suggest appropriate priorities and collaborations for current and future work.

The overriding sense of the conference was that educational research planning must emphasize focus and selectivity. The Board heard a consensus among conference participants that education research should be concentrating its inquiries on those areas that the public and the profession believe are important, as well as anticipating problems that will become important. Among the conferees, the appropriate topics were identified as reading and language learning; expanded attention to mathematics; the dynamics of teacher performance and effectiveness in schools and classrooms; and new emphasis on technology and telecommunications; international studies; and learning in family, community, and workplace settings.

Student Achievement

Members of the Board believe that the focus of research attention must be narrower still, in order to concentrate on something both important and possible.

Recommendation 1: Student Achievement — The priority for research in education must be high achievement for all students and, within that domain, the initial emphasis should be on reading and mathematics achievement.

Raising student achievement is a priority for education supported alike by parents, business leaders, public officials, and educators. But to attain high achievement for all students requires success in combating the most difficult and challenging issues of student performance across America. These are issues sharpened for us once again, recently, by international comparisons from the TIMSS in which both the strengths and the shortcomings in achievement among our youth are apparent. TIMSS data indicate that our younger children, age 9, demonstrate mathematics and science knowledge and skill at levels approximating those of children in other economically developed countries. But as they progress through the school system, they fall farther behind, so that by 12th grade, American students are among the lowest scoring students in the study. Before the TIMSS results were released, we may have been able to take comfort that our most able students ranked with those of other nations, but that has now been disproved as well. The TIMSS results repeated findings from the National Assessment of Educational Progress (NAEP) and other sources, showing that achievement of children from minority backgrounds and from low-income families, on average, continues to lag far behind that of the majority population.

These characteristics of student achievement—that (1) it is below levels experts believe necessary for maintenance and preservation of American democracy and for full participation in a vibrant economy in the 21st century, and (2) that there are unacceptably wide gaps across members of our population—are longstanding and have resisted well-intentioned attempts over many years to remedy. Members of the National Educational Research Policy and Priorities Board believe that the combined efforts from researchers across several disciplines, developers, and teachers can make a difference if student achievement is the priority.

We have called on the National Academy of Education (the Academy) to provide assistance in forming a research agenda around high student achievement, one that would build on what is already known, and one that would capture the most promising areas for further exploration. The Academy was asked to create a possible agenda on a series of focused research questions. The Academy has suggested three strands of work. The first strand is research on learning, especially across transitions in children's lives; the second on teaching in relation to learning as a professional practice to support student learning; and the third, strengthening the links between educational research and the practice of schooling. The first two are discussed in this section on priorities and problem-centered activities. The third is discussed below under collaborative work because it is directed toward constructing more powerful methodologies for conduct of research in education—not only in the area of high achievement for all students, but other topics as well.

The ability of the United States to make substantial progress toward the goal of high achievement for all students is limited by assumptions about the nature of research, learning, and teaching that cause policymakers and practitioners to neglect important complexities associated with education. It is, for example, usually assumed that the results of researchers' investigations should have important practical implications, whether or not the researchers are trying to improve educational effectiveness. Regarding students' learning, students are usually assumed to learn procedures and facts independently of their comprehension of the concepts and principles that make them understandable, and students' learning in school is assumed to occur independently of their abilities and personal identities outside of school. Teachers are assumed to develop skills and subject-matter knowledge independently of the social and cognitive challenges they deal with in their classrooms as they interact with students. In fact, however, the relations among all these factors must be better understood, and to do so, education must be thought of as a complex professional undertaking.

Transitions

In the first strand of its measures for focusing research on high achievement for all students, the Academy has advised us that the most critical and promising research questions fall into two areas. One involves transitions that students must accomplish in order to progress successfully through the school curriculum. The other concerns transitions that involve the social organization of learning in schools and its relation to the activities of students outside of school.

The Board recommends that research on teaching and learning in school be focused on critical transitions that include important developments in conceptual understanding as students move through the school curriculum, and as they move between the school and other communities. To achieve at high levels, students must succeed in critical transitions that require mastery in general aspects of knowing and understanding that are often not explicitly taught. The expectations in school for these general aspects of understanding and learning do not match with the experiences of all students, and the transitions are much easier for students for whom the school routines and practices are in close alignment with those that prevail at home. The difference is generally unfavorable to students of low-income families.

Recommendation 2: Reading, second language learning, and mathematics — Recent reports from the National Research Council, Preventing Reading Difficulties in Young Children, and Improving Schooling for Language-Minority children: A Research Agenda, synthesize strong bodies of research knowledge. A similar study on mathematics is currently under way. In each case, research is needed now to analyze how the results of our knowledge can be implemented in school programs and what factors lead to success and difficulties. In reading, research is needed on how students become facile at reading complex text as they transition to advanced academic subjects such as history, social science, mathematics, and science. Research in both short- and long-term effects of specific education interventions for English language learners is needed, as well as techniques of assessment to measure competence, and transition points (a) from the first oral language to English, (b) from oral language to literacy, and (c) from literacy to the academic discourse of specific disciplines. In mathematics, research is needed on why students have so much trouble making transitions (e.g., from concrete objects to abstract ideas), understanding formal representations, multiplicative reasoning, and essential mathematical and statistical concepts such as chance, randomness, and probability.

Recommendation 3: Organization for learning out-of-school — To take advantage of learning environments in which children from impoverished backgrounds often display more competence than in school settings, research is needed to design and test different models of after-school and summer programs to motivate, engage, and benefit low-income children. Work is also needed on types and features of after-school opportunities that most effectively motivate academic achievement and positive self-estimations; and how to design and test different models of collaboration between schools and community groups dedicated to providing strong learning environments for disadvantaged children.

Recommendation 4: Organization for learning in-school — Retention, pull-out remediation, tracking, and segregated special education programs that stratify by race, class, and gender opportunities to learn do not result in high achievement for all students. A more complete inventory of knowledge about effective practices for teaching academically challenging curricula with groups is needed, both for school populations in general, and for heterogeneous groups in particular. Research is needed on questions of time for children to master challenging curricula, supportive school structures, and expectations for the breadth and depth of content. Within each of these is the question, do students from middle-income families as well as students from low-income, ethnic, and linguistic minority backgrounds benefit from each organizational practice? An important area of inquiry is whether there are academic benefits to classroom diversity—does diversity improve subject-matter learning? (Note: Recommendation #5 addresses teaching practice aspects of school organization for learning.)

Teaching and Learning

The second strand of research to promote high student achievement in reading and mathematics is teaching as a professional practice. Without improving our understanding of what it will take to produce a well-prepared and professional corps of teachers, school improvement will not be possible. Students living in poverty and ethnic minorities have been historically underserved by American educational institutions and are an increasingly large proportion of the student population. No one doubts that teachers will have much to learn in the years to come in order to be successful in helping all students reach high levels of achievement. There should be a particular concern with producing new knowledge about connections between professional development and improving education for currently underserved populations; namely, children and adolescents whose experiences and dispositions do not match with the expectations and social organization of schools.

Recommendation 5: Linking changes in teaching practice with improved student learning — Information is needed that can guide teachers and institutions who want to change their educational practice, particularly to reduce inequities in the opportunities of students who differ in socioeconomic status, ethnic background, and gender to learn successfully. This is especially important regarding the achievement of deep intellective competence advocated in current educational reforms. Such research would examine fundamental issues about the nature of teaching and learning, including, but not limited to, the importance of the skills and knowledge of teachers. Expansion is needed in our knowledge and understanding of teaching practices, including teaching tools such as assessment, that are successful with students who bring different cultural resources to their own and to other students' learning. Research would examine, much more than past research has done, issues of what it takes to do effective and successful teaching with diverse populations of students. (Note: Recommendation #4 addresses the school organization context in which effective teaching for student learning takes place.)

Recommendation 6: Linking teachers' professional development and teaching practices — Research is needed to understand what effective teachers do and how they do it. Successful teaching involves not only the exercise of skills and application of knowledge but also flexible improvisational adaptation in classroom circumstances. Research is needed to understand the roles of more profound knowledge and comprehension of subject-matter concepts and methods, both at the level one is teaching and in relation to other disciplines and grade levels, as well as the role of understanding processes of students' learning. Research should also investigate how the structure of teachers' work supports or hinders their "on-the-job" learning and what kinds of abilities are learned in particular situations that can transfer to other settings with different circumstances.

Recommendation 7: Understanding and supporting successful professional development — There is need for a better understanding of teachers' development as "professional learning," and understanding teaching as a "learning profession." The prevalent model of learning how to teach—the knowledge goes in during teacher education or professional development and then comes out to be used in the teachers' own classrooms—does not account for the engagement of teachers themselves in improving the practice of their profession. What teachers need to learn to put reforms in place is not separable from their actual teaching practices or from the development trajectories of their careers. Research must examine ways in which people of diverse cultural, ethnic, and socioeconomic backgrounds are attracted to careers in teaching and how professional development resources can help increase and maintain diversity in the teaching profession, while continuing to improve practice. Further research on teachers' communities of practice is needed, building on findings that norms of responsibility and collegial efforts at professional problem solving are the most critical factors in improvement of teaching and learning.

Focusing the research agenda as the Board has suggested means that some important topics will not receive much attention. For example, such areas as cultural and political contexts of schools, educational policy, and school finance; education governance; and learning environments and educational technology—all of them areas in which important work might be done if sufficient resources become available—would not receive significant attention under the Board's view of priorities.

Among all the topics that would be deferred under the Board's identification of priorities to achieve high achievement for all students, one that the Board would single out as a candidate for inclusion at the earliest opportunity is assessment of teaching and learning for purposes of accountability.

Accountability is an increasingly important issue in educational research and practice. Current accountability measures, however, do not match the goals of most educational reforms for students of low-income families, especially those reforms aimed at improving their complex thinking and participation in activities of inquiry and understanding. One issue to be considered is the limitations of norm-referenced tests that are conventionally presented in standardized, multiple-choice formats. Criterion-referenced measures, aligned with teaching and learning standards, may assess the competence of these students more productively than norm-referenced tests. Alternative methods that are responsive and valid guides for instruction of students who come from a background of poverty should be developed and studied as soon as adequate funding can be attained.

High Standards of Quality Are Created and Upheld

The single criterion by which any scientific enterprise must be judged is the quality of its work. Scientific norms must be known and shared. The expectations for explicit hypotheses, sound designs, appropriate measures, sufficient data of good quality, and logical analyses must be widely shared. High standards must be insisted upon in all areas of a scientific agency's work—in selection of proposals, design of appropriate methodologies, creation of research agendas, identification of effective and promising practices, and evaluation of all efforts it conducts or supports.

The primary means by which high standards have been developed and assured in federal agencies has been through extensive networks to assure involvement of peers. Peer review is much more than a bureaucratic instrument. It is a major vehicle of communication between the government and the field, a process through which principles about research priorities and technical quality of research are clearly articulated, and applied to proposals. In the 1994 legislation, Congress made its intent clear that a peer system must be applied to every aspect of OERI's work. The law requires:

that a system of peer review be utilized by the Office for reviewing and evaluating all applications...which exceed $1 million;...evaluating and assessing the performance of all recipients of grants;...cooperative agreements and contracts;...and for reviewing and designating exemplary and promising programs...

such standards as may be necessary to govern the conduct and evaluation of all research, development, and dissemination activities carried out by the Office to assure that such activities meet the highest standards of professional excellence. In developing such standards, the Assistant Secretary shall review the procedures utilized by the NIH, NSF, and other federal departments or agencies engaged in research and development and shall actively solicit recommendations from research organizations and members of the general public.

In this assessment of operation of the standards in fiscal years 1996 and 1997 and in two types of competitions—field-initiated studies and Research Centers-- Board members learned that as many as a third of reviewers had not conducted research in education, even though that is a requirement in the standards. Among those who might have had research training and had themselves conducted research, that training and research experience was in broad topical areas related to the competition, but not necessarily in the methods and design of research in the proposals. In examining the reviews provided by OERI panelists, the Board-commissioned study found that most reviews provided little depth in their commentaries. Reviews were most detailed about project design and significance, least detailed on staffing, budget, and management plans. Applicants frequently disagreed with reviewer comments, saw the comments as superficial or irrelevant, found a lack of comments about design, and cited a lack of examples. Applicants also noted limited explanations and mentioned large discrepancies among reviewers. They believed that proposals had not been carefully read and said that comments were illegible.

Standing Panels

As a result of this review, the Board's principal recommendation on research quality is addressed to the establishment of standing panels.

Recommendation 8: Standing panels — Standing panels should be established to review proposals for each OERI Institute. These would be comprised of 25 to 30 members, but with some overlapping membership, so that problems that cross boundaries can receive informed attention and so that members of one panel with special knowledge could be invited to serve on another Institute panel.

This is the Board's most urgent and important recommendation on peer review processes, and it can be implemented by OERI on its own authority both easily and relatively quickly. Standing panels, frequently used by such organizations as NIH, offer the most compelling mechanism the Board could find to improve the quality of the review process. Standing panels provide continuity from the announcement of government funding opportunities to the decisions on proposals to fund. They provide an informed group to build areas of research over time so that the results are cumulative rather than episodic. Such panels can attract experienced members who will agree to serve, because the repeated contacts with colleagues are more professionally rewarding than membership in ad hoc settings. They can provide a forum where the accumulating knowledge can be sifted and interpreted, and new lines of research can be identified.

Thus, standing panels are a device to attract the very people whose judgment is needed to ensure that research proposals are of the highest scientific merit and are addressed to high priority national education needs. They can also play a crucial role in guiding and evaluating the direction of research.

The Board has additional recommendations that complement and support the standing panels.

Recommendation 9: Panel membership — Panels should represent a broad range of perspectives. They must include members with strong disciplinary and methodological expertise. Across OERI's panels, gender, race, ethnicity, and geographic diversity must be respected. Panel members should be nationally recognized figures.

Recommendation 10: Standards for panelists — The Board continues to support peer review standards adopted by OERI, with Board approval, which specify that all reviewers meet three criteria: "(a) demonstrated expertise, including training and experience, in the subject area of the competition; (b) in-depth knowledge of policy or practice in education; and (c) in-depth knowledge of theoretical perspectives or methodological approaches in the subject area of the competition."

The Board has made further recommendations to the Assistant Secretary to enhance the effectiveness of reviewers, reduce workload for reviewers and applicants, and improve center competitions. For all grant competitions, it is important that the quality of research designs be rated by reviewers with appropriate technical expertise. The Board strongly prefers that each proposal be read by a minimum of five people. More logistical and other support should be provided for reviewers along with more in-depth training, and better formats should be created in the technical review form to guide the reviews. Reviewers should be expected to provide specific, but not necessarily detailed, feedback to applicants. Reducing workload for both reviewers and applicants would enhance the likelihood of accomplishing these changes. For example, the Assistant Secretary should consider making grant announcements and appointing submission dates earlier in the fiscal year; reducing the number of full applications through use of preliminary applications; reducing the number of pages permitted for center applications and the page limit for attachments; assignment of specific primary, secondary, and tertiary reviewers to applications; and conduct a small pilot project to determine how technology might be used to support the peer review process.

For center competitions specifically, the Board has urged the Assistant Secretary to clarify the project design criterion so that reviews address the end projects proposed as well as the overall center design; increase weighting for management and clearer instructions; and provide planning grants. These changes require modifications in regulations.

The Board also has recommendations for enhancing peer review processes that may require changes in the 1994 Act.

Recommendation 11: Distinguishing between field-initiated and directed competitions — OERI should distinguish between field-initiated competitions and those that are directed, rather than trying to combine the two.

Recommendation 12: Funding for peer review — The allowable percent of funds to support peer review should be increased so that the necessary standing panels may be established and logistical support provided.

Recommendation 13: Definitions — The term "research" should have a narrower definition than it has in the 1994 law so that the boundaries of focused competitions for research can be limited. Research should encompass basic research in education as well as investigations, experiments, and inquiry to develop new knowledge or apply tested knowledge. It should exclude development, planning, and demonstrations. The term "national significance" needs to be clarified through regulations or in legislative language so that reviewers understand that it includes research opportunities, not only important problems identified by educators.

[Findings]

[Goals & Recommendations (part 2 of 2)]