A r c h i v e d  I n f o r m a t i o n

2.0 Even Start Evaluation Requirements

Even Start was initiated as a demonstration program. After its first decade, it remains a program that is designed to help demonstrate and learn about the best ways of implementing and delivering family literacy services. To assist with this mission, strong evaluation requirements have been a prominent and continuing feature of the Even Start legislation. Here we briefly discuss Even Start's evaluation requirements at the national, state, and local levels.

National Evaluation

The most recent comprehensive Even Start legislation is the 1994 Improving America's Schools Act, Part B of Title I of the ESEA. That legislation called for:

"...an independent evaluation of programs assisted under this part—
(1) to determine the performance and effectiveness of programs assisted under this part; and
(2) to identify effective Even Start programs assisted under this part that can be duplicated and used in providing technical assistance to Federal, State, and local programs." (P.L. 103-382, Sec.1209).

To carry out this legislative authority, the Department has supported an ongoing national Even Start evaluation in which each local project uses a PC-based management information system to provide the Department with descriptive information on program operations, services offered, participant characteristics, participation rates, and program outcomes; and in which selected projects participate in more intensive studies of program impacts. The national Even Start evaluation is supported with funds reserved from each year's appropriation for the Even Start program.

State Evaluation

The Even Start legislation does not call for state-level evaluation, and Even Start does not provide any funding specifically for state-level evaluations. However, Even Start is administered by state agencies and in the past few years, several states have assumed the responsibility for planning and conducting evaluations of the projects that they have funded. To date, state evaluations generally have been funded either by using state administrative funds, or by local projects combining their evaluation funds to provide a large enough pool to conduct coordinated local project evaluations that also can serve a satisfactory state-level study purpose. Though our current understanding of the nature of state-level studies is limited, we do know, for example, that some states are putting their own management information systems in place, others are conducting pre-post evaluations in each funded project, and still others are working with grantees to devise a mutually-agreeable common set of measures.

When we inquired about the availability of evaluations, few states had reports available for public release. However, many state coordinators said that they either were planning evaluations or were in the process of conducting them. These state-level studies typically were being performed through contracts with independent contractors.

Recent amendments to the Even Start program in 1998 include a requirement that states develop results-based indicators of program quality and to use these indicators to monitor, evaluate, and improve Even Start programs. The Department will be able to work with states in developing their indicators of quality through the Statewide Family Literacy Initiative grants funded through the same set of amendments to Even Start.

Local Evaluation

In addition to language requiring a national evaluation, the most recent Even Start legislation contains the following language about local evaluations:

"Each program assisted under this part shall...provide for an independent evaluation of the program." (P.L. 103-382, Sec. 1205).

When Even Start was administered at the federal level (from 1989 to 1991), it was recommended that local projects allocate $5,000 to $10,000 per year to cooperate with the national evaluation and to conduct an independent local evaluation. This rule of thumb for the amount to be spent on a local evaluation has been maintained in some states, while other states offer little or no guidance on the amount of funds to be spent on a local evaluation.

Assuming an average expenditure of $5,000 per project per year, the total amount of federal funds spent on local evaluation activities has grown from about $380,000 in 1989 when there were 76 Even Start projects to more than $3.25 million in 1997 with about 650 projects. This is a substantially larger amount than what has been spent on the national evaluation during the past decade (about $500,000 to $750,000 per year).

While the Even Start legislation does not provide local projects with much direction on how to spend their local evaluation funds, the Department has provided several forms of guidance about local evaluations to Even Start project directors. Further, since Even Start is now a state-administered program, each state can provide its own advice to local projects on how to best comply with the local evaluation requirements.

As part of the first national evaluation (1989-90 through 1992-93), the Department asked the national evaluation contractor to work with local projects so that they would be able to submit evidence of effectiveness to the Department's Program Effectiveness Panel (PEP). Consequently, sessions on preparing PEP submissions were held at national evaluation meetings and interested projects were trained in how to use national and/or local data to make a case for the effectiveness of their projects. In response to questions raised by local project directors at national evaluation meetings, Department staff responded that while local evaluation monies were intended to fund a local study that would complement the information collected for the national evaluation, local evaluation funds also could be used to pay for various tasks associated with complying with the national evaluation, e.g., interviewing parents, entering data into the computerized MIS, and attending national evaluation conferences. In these early days of Even Start, many local projects were uncertain about the purpose and requirements of their local evaluation.

As part of the second national evaluation (1993-94 through 1996-97) the topic of how to conduct local evaluations continued to be raised and discussed at national evaluation conferences. For example, the June 1994 Even Start Evaluation Conference contained a session titled "Two paths to program validation: PEP and the National Family Literacy Dissemination Project." This built on the emphasis seen in the first national evaluation on having local projects prepare and submit evidence of their effectiveness to the Department in order to possibly obtain additional funding as a validated approach.

Another session at the 1994 meeting was titled "Designing and implementing a local evaluation." At this session, staff from RMC Research Corporation worked with participants to design a "vision for local evaluation guidance" that formed the basis of a Local Evaluation Guide for Even Start projects (Dwyer, 1998).^[7] The vision statement specified that projects were to conduct "comprehensive local evaluations," that among other things assess "effectiveness in terms of participant outcomes," and that are "complementary to the national ESIS reporting system" (Exhibit 1).^[8]

In the third national evaluation (1997-98 through 2000-01) there has been a continued emphasis on the use of national evaluation data for local purposes, as evidenced by the annual preparation of Project Profiles which compare local descriptive and outcome statistics to data from state and national aggregates. The emphasis on gathering information for PEP submissions disappeared^[9], replaced in part by an Observational Study of Even Start projects that provided evidence of positive outcomes over at least a two-year period.

The most recent Even Start guidance provided by the Department contains the following language:

"Each Even Start project is required to provide for an independent evaluation of the project. See section 1205 (10) of the Act. These evaluations provide local projects, States, the Department, and the Congress with objective data about the activities and services provided by the project, the participants served, the retention rates of those participants, and the success of the families in the project." (Department of Education preliminary guidance).

The Department interprets the requirement for an independent local evaluation to mean that the evaluation must be done by a person not directly involved in the day-to-day administration of the project. Thus, the evaluation could be done by an evaluator who is external to the LEA, or by a person within the LEA who does not work directly for the Even Start project.

Other related resources available to local projects include a Guide to Quality for Even Start Family Literacy Programs and a Program Self-Assessment. Prepared for the Department of Education by RMC Research Corporation, the Guide provides projects with a discussion of the components of an Even Start project and ways to ensure the quality of those components. Projects are supplied with a self-assessment measure that they can use to rate the quality of their projects. While projects are not asked to report on the results of the self-assessment in their independent evaluation reports, we have heard that projects do use the Guide and Self-Assessment in analyses of their programs.

Exhibit 1 "Vision for Local Evaluation Guidance" (quoted from June 1994 evaluation conference materials) "The Department of Education expects all Even Start projects to put into place soon comprehensive local evaluations that yield information useful for the ongoing management of programs, provide the basis for program improvements, and meet accountability demands of funders and collaborators. For programs that have attained operational stability, local evaluations should also address assessment of effectiveness in terms of participant outcomes, including those outcomes that can be compared with national results. The Department of Education requires Even Start projects to conduct local evaluations in addition to being active participants in the national evaluation of the program. Local evaluation data collection should be complementary to the national ESIS reporting system, taking advantage of required national data for local uses. Careful planning of an information system is a key part of local evaluation design to facilitate maximum use of the national evaluation tools and database results. Because all projects are not currently required to provide outcome data, it is important for the future development of Even Start to encourage, document, and share what is known about an array of program effects. Local evaluations offer the opportunity to experiment with measurement of the wide array of short, intermediate, and long-term outcomes that result from Even Start programs. Information about participant outcomes measured in a variety of ways has great importance for depicting the value of Even Start interventions to decision makers. The local data is needed to augment what is collected via ESIS. For projects that yield exemplary outcomes, there are two national avenues of dissemination, the Program Effectiveness Panel and the National Family Literacy Project. Approval by either dissemination mechanism requires systematic collection of implementation and outcome data. For all the above reasons, planning the design of a local Even Start evaluation involves numerous choices and a high degree of communication between project personnel and evaluation staff. Choices made early on in the planning will affect the ways in which comparisons can be made later to gain both power and efficiency for local efforts. Anticipation of eventual uses for information pays off later. To encourage thoughtful planning, the Department of Education (with the help of experienced Even Start evaluators) is preparing a guide for designing local Even Start evaluations. The guide will function both as a planning tool for local project directors to use in dialogue with potential program evaluators during program start-up and also provide guidance for evaluators during the design phase. The types of evaluation questions and outcomes posed by the guide, the suggested linkages to the national evaluation, and the possible sources of relative comparisons will provide the basics for taking advantage of local project data. We anticipate that SEA monitors will also find the guide helpful in their support of local project development and improvement. Further, the guide will help SEAs use the local project data they will receive as part of the ESIS system."

⁹ The National Diffusion Network and Program Effectiveness Panel lost their funding during this period.

-###-

[1.0 Overview of Even Start]

[3.0 Study Methods]