A r c h i v e d I n f o r m a t i o n

Mapping Out the National Assessment of Title I: the Interim Report - 1996

Section 7:
Plans for Evaluating Title I

The evaluation agenda for the National Assessment of Title I (NATI) focuses on addressing the questions raised by Congress and by the Independent Review Panel, as outlined in preceding sections of this report. The scope and details of the evaluations planned have been guided by the need to report on performance indicators at various levels of governance that support Title I and educational improvement--from the classroom, district, state, and federal levels. Much greater collaboration is anticipated across levels of government to conduct evaluations and data syntheses useful for benchmarking progress. The agenda for evaluating Title I will draw on research studies and student assessments supported by the U. S. Department of Education and on evaluations conducted by other federal agencies, state and local education agencies, and private foundations with initiatives that involve Title I schools. The NATI will report findings to Congress and to the field on a schedule that is timely and supports continuous feedback.

The recognition that Title I operates within a larger context of school improvement has broad implications for how evaluations are conducted. Evaluations will need to actively involve state and local stakeholders in developing studies that have meaning and usefulness beyond national reporting on Title I. Participation by state and local policymakers, practitioners, and parents on the Independent Review Panel is a useful start, but the Department will need to reach out further to broaden collaboration.

The purposeful alignment of the key features of Title I with state and local school improvement efforts makes this collaboration possible, and indeed essential. New technologies will help facilitate the sharing of information on promising strategies across levels of government and with principals, teachers, and parents at the school level. The Department supports the creation of an electronic network of schools engaged in schoolwide programs as a first step toward developing more integrated, user-friendly approaches to tapping information on school progress in implementing Title I reform. The Department intends to reciprocate by supporting the sharing of information on promising strategies and supports for improvement across the network.

The NATI's evaluations also must be informed by a solid and expanding research base. It is critical that the Department focus support for research on the most promising approaches for improving learning for students at risk of school failure and on the conditions that support improvement. These findings can help frame evaluation issues, in addition to informing practice. At the national level, the mandated longitudinal study will provide rigorous measures of the impact of standards-based features in Title I on improving curriculum and instruction and consequent student performance. Student achievement information collected through the longitudinal study will be supplemented by analyses of student performance at national and state levels drawn from the National Assessment of Educational Progress (NAEP). The Department's work with the National Science Foundation will also allow for joint reporting on the effects of systemic reform on the math and science achievement of at-risk students in relation to other students. At the state level, analyses of state assessment information will examine the progress of Title I students in meeting challenging standards particular to individual states. The NATI will also analyze data reported by districts to the extent that such information is available. The Department anticipates much closer collaboration across research centers, laboratories, and comprehensive assistance centers to conduct this work, as well as across research, evaluation, and program offices within the agency.

The NATI will report on the performance of Title I throughout its authorization cycle, not just through a final report prior to the program's reauthorization. Performance indicators for Title I will be reviewed when justifying yearly budget justifications and determining priorities, in annual reports to Congress and the general public, and as new information becomes available on the program's progress.

This section describes major planned studies to measure the progress and impact of Title I within the broad context of reform, including links among studies and with other federal initiatives. These studies include a performance indicator system and baseline surveys, school- and classroom-level studies, studies at the state and local levels, and federal studies.

Performance Indicator Framework

A performance indicator system provides the general framework that links evaluations to information needs. As recognized by the Government Performance and Results Act, performance indicators contribute to improvements by pointing out problems in implementation, highlighting promising strategies, and helping to maintain a focus on better results for students. The development and tracking of performance indicators is an ongoing activity. The Department also seeks to help states, school districts, and schools develop their own performance indicator systems for self-assessment. Specifically:

Indicator Development, which involves a review of state plans, progress reports, Department performance monitoring reports, in addition to evaluative studies to select key indicators to measure progress and shortcomings. Title I data on the numbers of students participating, schools choosing schoolwide programs, and schools identified for improvement will be routinely collected through annual state reporting, supplemented by more in-depth information compiled through program monitoring.
Pilot Efforts to Support States in Collecting and Analyzing Performance Indicators, which will encourage efforts in a sample of states to improve the quality, scope, and usefulness of the collection and analysis of performance indicators related to the implementation of Title I, other reauthorized ESEA programs, and Goals 2000. These federal/state partnerships are intended to enhance the use of data for program accountability and improvement while building upon data that individual states collect for their own continuous improvement. Pilot efforts will address the varying needs for indicator information at each level. The Department intends to begin with a consortium of states that volunteer to work on developing data useful to state and federal monitoring of progress. The intent is to move beyond the pilot sites to larger efforts that draw upon the states' own data systems for Title I, other ESEA, and Goals 2000 program reporting.

School- and Classroom-Level Studies

Data collection efforts at the school and classroom levels include:

Baseline surveys of school principals and teachers, which will provide the first indicators in the information system, offering a current snapshot of school-based perceptions of federal, state, and locally supported reforms and the extent to which reform efforts have begun to influence changes in staff professional development, a focus on higher standards for all students, classroom practice, and parent involvement.
The surveys will be implemented in winter 1996 through the National Center for Education Statistics Fast Response Survey System. A special survey of principals on parent involvement will ask about the use of parent/school compacts, staff training, and parent involvement policies in Title I schools, with a particular focus on Title I schoolwide programs that are designed to engage all parents in whole-school improvement. Data from this study are expected by fall 1996. A follow-up study is planned in FY 1997.
Information Networks with Title I Schools, which will support the creation and maintenance of electronic networks of schools engaged in Title I reform. The intent is (1) to tap information on these schools' progress and (2) to share information across the networks and with technical assistance providers on schools' problems, needs, and successes in implementing Title I and school reform. The Department plans to begin by setting up an electronic network for Title I schoolwide programs and Title I councils and teachers.
An Omnibus Longitudinal Evaluation of School Change and Performance, as separately mandated by Congress under IASA, Section 1501(c), will analyze the cumulative impact of standards-based federal support for upgrading educational performance in Title I schools. To the extent that resources permit, the study will focus in particular on Title I schoolwide programs and schools chronically identified for improvement. A longitudinal design will provide a better understanding of how schools and classrooms change within the context of federal, state, and local standards-based reform. As proposed, the study will examine changes in student performance and key factors related to high and low performance, through a mixed approach involving national assessments such as NAEP, and state and local exams. Resource constraints will likely preclude the use of a nationally representative sample of schools or, as in typical longitudinal studies, tracking the performance of individual students over time. Instead, the evaluation may focus on a limited number of states, with a more in-depth examination of curriculum and classroom instruction at the elementary school level. The Department has awarded a design contract for this study and will award a competitive procurement to conduct the study in FY 1996.
The Study of the Impact of Title I Schoolwide Programs on Migrant Children, as separately mandated by Congress, will examine the extent to which schoolwide programs affect learning opportunities for migrant students. Because schoolwide programs enable all students to benefit from Title I resources, regardless of when they enter a school, those programs may facilitate access to services for migrant students. At issue, however, is the concern that the unique needs of migrant students may not be met through schoolwide approaches. Given the limited English proficiency of many migrant students, the study will focus on how Title I accommodates their need for language instruction. The Department plans to award a competitive procurement in FY 1996.

State and Local Studies

Current and planned studies at the state and local levels include:

The Study of States' Planning and Early Implementation of Goals 2000 and ESEA, which will provide baseline indicators of the planning process and early implementation of Goals 2000 and ESEA programs, particularly Title I. The evaluation will focus on how the legislative framework and federal resources are incorporated into the context of state school improvement efforts. Key issues will address state activities, including the process of developing state plans, setting standards, and aligning assessments with higher standards in the basics and core subjects. State-level support for school improvement will also be a focus, including the various ways states are providing professional development and technical assistance to districts in planning, performance accountability (including incentives and sanctions), and other supports (such as waivers) to encourage local flexibility and innovation. In addition to Title I, other programs included in this first look at state-level planning and implementation are Goals 2000, Title II (professional development), Title IV (safe and drug-free schools), Title VI (Innovative Grants) and the McKinney Education for Homeless Children and Youth program. Case studies will be conducted to examine the various stages in design and implementation among the states. Findings are anticipated in early 1997; a follow-up study is planned for FY 1997.
The Study of Family Involvement in Title I, as mandated by Congress, will draw on two special studies--one focused on state policy and the other on school strategies to engage parents in their children's learning. The Council of Chief State School Officers will conduct an analysis of state policies promoting family involvement to examine how states encourage districts and schools to engage parents in setting policy, train school staff and parents to work together, and uphold mutual responsibilities for learning. A survey of a nationally representative set of schools about family involvement policies and practices will collect information on their initial experiences in carrying out the new Title I requirements. The survey will compare the gap in the degree of family involvement occurring between high- and low- poverty schools; replication of the survey will examine whether the gap is closing over time. Work on both studies began in fall 1995. Results from the state study are expected by spring 1996; results from the school study are expected by late fall 1996.
A Study of Local ESEA and Goals 2000-Supported Planning and Implementation, which will analyze districts' efforts to implement ESEA programs, particularly Title I, and Goals 2000-supported efforts within the context of reforms. Key issues to be examined include: (1) districts' progress in consulting with schools on strategies to align curriculum and professional development to achieve higher standards developed under Goals 2000 or Title I, (2) the use of federal funds to address the needs of target populations, (3) the implementation of school improvement requirements, and (4) coordination across programs. A competitive procurement is planned for FY 1996, with findings available in 1998.
An Evaluation of Title I Participation of Private School Students, which will survey a nationally representative sample of districts to examine the impact of changes in allocation procedures on participation of private school students. The survey will examine 1995-96 as the transition year during which districts must implement new procedures for determining eligibility for funds. Basing allocations on the number of low-income students, not on the number of educationally disadvantaged students, is expected to have an impact on private school student participation in Title I because private schools are less likely to participate in the federal free or reduced-price lunch program and may lack other means for assessing the income of students.
An Evaluation of Within-District Targeting in Title I, which will examine resource issues including how districts allocate Title I funds to schools, what poverty data districts use to determine eligibility, and which exceptions districts make to the rules governing allocations. Special attention will be given to (1) allocations for high schools and middle schools, (2) the level of Title I funding in schoolwide programs compared to targeted assistance schools, (3) the effects of the minimum allocation rule for Title I schools, (4) the extent to which waivers are used to provide Title I funds to schools that would not otherwise be eligible, and (5) the effects of any reductions in Title I funding. The study is expected to be awarded in spring 1996 and to provide preliminary data for the NATI by fall 1996.
A Joint Study of the Criteria for Alignment of Content Standards/Curriculum Frameworks and Students Assessments, which will help state and local educators measure the alignment of standards/curriculum frameworks and assessments and improve understandings of current approaches to alignment. The study will examine current definitions of alignment, describe effective approaches to define and measure alignment in current practice and analyze current and proposed approaches to alignment, using an expert task force. The study may also report to states and districts recommendations for criteria of alignment.
This National Science Foundation-supported study is part of a multi-year analysis of the state curriculum frameworks and standards in mathematics and science by the Council of Chief State School Officers and is an activity of the National Institute for Science Education at the University of Wisconsin-Madison. Analysis is expected to be complete by fall 1996.

Federal Studies

Title I studies at the federal level include:

The Evaluation of Federal Efforts to Assist in School Reform, which will collect indicators from the customers' perspective of the federal government's processes and performance in promoting improved state-, local-, and school-level practices in implementing federally supported reform efforts, especially under Title I and Goals 2000. This study will address congressional mandates to evaluate federal assistance to states, focusing on the role and effectiveness of the Department's communications, technical assistance, issuance of regulations, reviews of plans, and other efforts--building on performance indicators and broader studies.
The study will analyze information from education personnel at the state, district, and school levels in order to assess the extent to which federal information and assistance are adequate, consistent, and useful in planning and implementing Title I and other elementary and secondary programs. It will also inquire about sources of information and technical assistance and their usefulness to policymakers and practitioners. In addition, the study will focus on the review of waivers requested under new waiver authority, to examine the process for approval and implementation, the adequacy of protection for target populations, and the scope of the flexibility proposed in the requests. A task order contract has been awarded. Findings are anticipated in early 1997; a follow-up study is planned for FY 1997.
The Study of Intercensal Poverty Data, as mandated by Congress, which supports the examination of district-level data developed by the Bureau of the Census that will be the basis for Title I allocations beginning in 1999. Congress has mandated that the Department contract with the National Academy of Sciences for this analysis. Preliminary work will begin with meetings in early 1996.

Studies of Related Programs

Several studies of related programs will provide information on Title I:

The Evaluation of Title II Professional Development, which will examine the quality and impact of professional development supported by the Title II Eisenhower program; its alignment with broader educational reform efforts of states, districts, and schools; and its impact on teachers' classroom practice. The study will focus on coordination of Title II with Title I in high-poverty schools. The Department anticipates conducting this evaluation in coordination with teacher enhancement studies supported by the National Science Foundation and the National Science and Technology Council. A competitive procurement is planned for FY 1996, with findings available in 1998.
A Review of National and State Assessments of Educational Progress, which will evaluate the National Assessment of Educational Progress, focusing on the developmental state assessments and developmental student performance levels. As mandated by Congress, the evaluation will examine the quality and usefulness of the state assessments and performance levels in conveying results on student achievement to the public. The study will also examine the usefulness of the National Assessment of Educational Progress in gauging the performance of Title I. A sole-source contract has been awarded to the National Academy of Sciences for this three-year study.

-###-

[Section 6: Effective Targeting of Title I Resources]

[Glossary]