A r c h i v e d  I n f o r m a t i o n

Synthesis of Local and State Even Start Evaluations

Background

The Even Start Family Literacy Program is a national program whose goal is to help break the cycle of poverty and illiteracy of low-income families with limited educational experiences. All Even Start projects build on existing community resources to integrate adult education, parenting education, and early childhood education services into a unified program. In addition to the Congressionally-mandated national evaluation, each Even Start project is required to provide for an independent local evaluation. Many states are also starting to conduct their own evaluations.

The Planning and Evaluation Service contracted with Abt Associates to examine the quality and content of a set of state and local Even Start evaluations. Abt collected 118 reports from projects in 19 states (and 4 state evaluation reports). This synthesis provides a characterization of the set of reports received. Based on the methodological strength of the evaluations outcomes studies, 24 reports were chosen for a more detailed review. The report also provides recommendations for strengthening local evaluations.

Key Findings

• Local evaluation designs are influenced by multiple factors and thus the methodology and quality of local evaluation reports vary.
  • A significant number of the evaluation reports received used designs that did not seem to be of high quality, particularly with regard to the quality of outcome data reported.
  • Guidance, project maturity, available funds, and project needs each contribute to the evaluation design. Many projects respond to the legislative requirement by focusing on project outcomes to show success for future funding.
  • While another important purpose of a local evaluation is to help manage and improve programs, the reports that included that kind of focus were the exception. Anecdotal evidence suggests that local evaluators might provide more information to projects than they include in the local evaluation report, such as programmatic improvement recommendations.
  • The great majority of local evaluations are conducted by evaluators from outside the grantee agency. The typical local evaluator is a university-based consultant. Some local evaluators have advanced degrees in a relevant field while others seem to have fewer formal credentials.
  • Local evaluations could be improved if projects were encouraged to engage in a continuous improvement effort. Local projects need guidance on the use of data and evaluation for continuous program improvement. The Department-sponsored Observational Study of Even Start is facilitating and assessing the use of the continuous improvement model with a group of ten Even Start projects. Local projects could also use guidance on the amount of funds necessary to ensure a high-quality evaluation.

• Of the 122 studies collected, 93 percent contained implementation information and 77 percent contained some information on outcomes.

  • Almost all of the implementation studies included a description of the project structure and activities. Sixty-two percent of these studies included participation information. Self-report was the most common measurement approach.
  • While 93 percent of all projects reporting for the national evaluation reported that they assessed outcomes in their local evaluations, only 77 percent of the reports reviewed were found to include outcome information.
  • None of the reports reviewed used a control group to evaluate the impact of Even Start. This is not surprising given the level of funding available for local evaluation. Only 10 percent of the local outcome evaluations used a comparison group (e.g., families in a similar program, children in Head Start). The most common design was the one-group pre-post study, where participants are assessed as they enter the program, and once again at a later point in time. Thirty-one percent of the outcome studies used a one-group post-only design, where participants are only assessed at one point in time, often at the end of a school-year or after a set amount of time in a program.
  • Sixty-four percent of the outcome studies measured children's progress through a direct assessment such as the PreSchool Inventory (PSI), the Peabody Picture Vocabulary Test (PPVT), or the Preschool Language Scale (PLS-3). Thirty-eight percent measured children through parent report, 29 percent through teacher report.

  • Seventy percent of the outcome studies measured adults' progress through a direct assessment such as the Comprehensive Adult Student Assessment System (CASAS) or the Tests of Adult Basic Education (TABE).

• There was great variation in the type and number of outcome measures used across projects, but Even Start participants generally showed improvement on standardized measures.

  • Even Start children generally gained on several measures of child development. These gains were mostly consistent with those in the national evaluation.

  • Even Start children were generally seen as scoring within age-appropriate norms on standardized tests. Teacher ratings of behaviors, skills, and interactions generally increased over time and indicated that most Even Start children were ready for school.

  • School records showed that Even Start children have higher attendance rates than the general school population. Increased attendance rates were not found in the follow-up of children from the In-Depth Study in the first national evaluation, although Even Start children were found to be tardy less often.

  • Statistically significant gains, in the same range as the average gains in the national evaluation, were observed in almost all projects that administered the CASAS or TABE. In addition, projects that collected GED status data reported substantial numbers or percentages of adults attained a GED during Even Start.

  • Parents almost always had very positive perceptions about Even Start's effects on their lives and the lives of their children. This is consistent with parent reports done in case studies as part of the national evaluation. However, there were few, if any, significant changes over time on rating scales of social support, family stress, and depression. Similarly, while parents had mostly positive perceptions about Even Start's effects on parent/child interactions, local evaluations generally were unable to detect significant changes on rating scales of parent behaviors and knowledge.

[Return to Planning and Evaluation Service]