A r c h i v e d  I n f o r m a t i o n

6.0 Recommendations

The Department currently spends much more on local Even Start evaluations (more than $3 million per year) than on the national evaluation ($500,000 to $750,000 per year). If the mere existence of a local evaluation report is useful, or if it is useful for project directors to have a local evaluation that concludes that Even Start is a success, then local evaluations can be judged helpful to local projects. However, if the criterion for success is that a local evaluation should be used to help improve an Even Start project, then this review of more than 100 evaluations was not able to show that local studies provide much useful information, either to local project directors, to State Coordinators, or to the Department.

How can we improve this situation? Most important, in our view, is to provide clear guidance on the purpose of local evaluations. The current mix of guidance (to assess program outcomes, to address questions of local importance, to improve program functioning) needs to be streamlined and focused. We agree with the importance of the Department's recent move towards a modification of the current system in which local grantees would be encouraged to use local evaluations as vehicles for program improvement. Some specific recommendations for improving local Even Start evaluations and for enhancing the ability of Even Start grantees to conduct continuous improvement efforts are given below.

Change the Even Start legislation and preliminary guidance to refer to "local continuous improvement efforts" instead of "local evaluations." Terminology is important, and the past and current use of the term "local evaluation" has become synonymous with a study only of program outcomes, with the implication that local projects will be in trouble if they cannot demonstrate positive outcomes. Use of the term "local continuous improvement effort" would more clearly signal the understanding that Even Start is a difficult program to implement, and the intention that projects are expected to engage in a systematic, ongoing assessment of the strengths and weaknesses, as well as the outcomes, of their approaches, and to use evaluation data to improve their programs.

Provide guidance to state coordinators and/or local projects on the amount of funding that needs to be spent on a good local evaluation which focuses on program improvement. Without such guidance new projects have no idea of the amount of funds to allocate for this activity, and some do not include a local evaluation line item in their budgets. Suggesting a set amount or a percentage of the total project's budget that should be allocated to evaluation will signal the importance of evaluation and data collection to local program staff.

Help establish a community of local evaluators that can exchange information about useful evaluation approaches by making a mailing list of local evaluators available, by helping local evaluators exchange e-mail addresses, and by having sessions for local evaluators at annual Even Start conferences.

Provide training for local grantees in using data collected at the state and/or national level for continuous improvement at the local level. Local family literacy projects often regard participation in state-level and national-level studies as a burden that offers little or no return. State and national evaluation sponsors could help ensure local use of data by providing training to local staff in (1) accessing data collected for state or national studies, (2) understanding what those data mean at the local level, and (3) using those data to assess the performance level of their projects and to improve unsatisfactory performance.

Provide guidance to local projects on the use of data/evaluation for program improvement. Program improvement currently occurs through trial-and-error, and is based on the hard-learned experiences of program staff. This is one way of improving programs, but greater gains could be made by accompanying these personal, anecdotal methods with a data-based systematic assessment of program strengths and weaknesses. Such systematic continuous improvement efforts have been described by Haslam (1998) in the Observational Study of Even Start Projects, by Alamprese (1996) in studies of workplace literacy programs, and by Appel (1998) in her work with local Even Start evaluations.

The Department, in the Observational Study of 10 well-implemented Even Start projects, is facilitating and assessing the use of the continuous improvement approach based on the following principles:

Collaboratively set outcome goals for children and families. Local project staff, evaluators, and families enrolled in the program work together to set concrete outcome goals to be achieved.

Devise an intervention to achieve those goals. The family literacy intervention should be designed to achieve the specific outcome goals set in the first step. The intervention should be based on practices shown to be effective in prior research as well as on proven program quality indicators.

Set intervention thresholds necessary to achieve goals. An assumption underlying all family literacy programs is that if a family is to achieve their goals they must participate in the program for a sufficiently long period of time at a sufficiently high level of intensity. Program staff need to define, ahead of time, the minimum intervention threshold that they believe is needed for families to achieve their goals.

Assess progress toward goals with sound measures. Assessing progress on a periodic basis by using sound measures (i.e., measures that have adequate reliability and validity, that have a history of use in similar studies, that are available in appropriate languages, and so on) is the best way of determining whether families have achieved their goals.

Use evaluation to monitor program quality and results, and to target areas for improvement. A comprehensive evaluation will help program staff monitor the quality of their family literacy program. It also will help staff keep track of each family's level and duration of participation in the intervention and assess progress towards goals. Knowing the quality of each program component, the extent to which families have participated, and the degree to which they are making progress will allow program staff to understand why some families do not achieve their goals (e.g., they did not participate at a sufficient level, or program components were of low quality), to identify program components that need improvement, and to target resources accordingly.

[5.0 Evaluation Review]

[7.0 References]