A r c h i v e d  I n f o r m a t i o n

Chapter 2

Design of the Second Even Start National Evaluation

In March 1994, the U.S. Department of Education awarded the second national evaluation contract, which consisted of four related but distinct components:

• The Even Start Information System (ESIS) to collect data from all projects;

• The Sample Study to collect outcome data from fifty-seven selected projects;

• Site visits to Migrant Education Even Start (MEES) projects; and

• Site visits to Even Start projects for Indian tribes and tribal organizations.

This four-year study continued to monitor the implementation and effectiveness of the Even Start program through analyses of the same critical issues examined in the first evaluation. In addition to the national Even Start evaluation, other studies of Even Start funded by the U.S. Department of Education and other organizations are examining various specific issues affecting the Even Start program. In addition, all local Even Start projects arrange for evaluations of their projects by independent evaluators.

The components of the second national evaluation are summarized in Exhibit 2.1.

Evaluation Components	Target Population		Types of Data Collected		Data Collection Procedures and Years		Basic Research Questions
Second National Evaluation Activities
Universe Study	All Even Start projects and participants		Participant characteristics, services, implementation, costs, participation outcomes		Parent interview, project questionnaire   1994-1997		Who is served by the program and what services do they receive? Is the program reaching the appropriate target population? How well was the federal funding on the program spent? How many of the programs were well implemented?
Sample Study	Even Start participants in 57 selected Even Start projects		Outcomes: school readiness, adult literacy skills, home environment, program participation		Child and adult tests (PSI, PLS-3, TABE, CASAS), HOME Screening Questionnaire, family contact logs   1995-1997		How well does the basic Even Start model work? Do participants perform better on key measures than similar persons who do not participate? What are effective practices and programs?
Case studies of Migrant Education Even Start projects	Three Migrant Education Even Start projects		Special issues regarding program implementation, effective practices, applicability of the ESIS		Staff interview, observations of educational activities   1994		What issues are special to implementing Even Start programs for migrant families? What are promising practices? How well does the ESIS capture the unique features of these projects?
Case studies of tribal Even Start projects	Three tribal Even Start projects		Special issues regarding program implementation, effective practices, applicability of the ESIS		Staff interview, observations of educational activities   1995		What issues are special to implementing tribal Even Start programs? What are promising practices? How well does the ESIS capture the unique features of these projects?
Additional Evaluation Activities
Focused studies sponsored by the federal government and other organizations	Projects selected using various criteria depending on the focus of the study		Focused assessment of specific program implementation and effectiveness issues		Many evaluation approaches used		Different research questions are addressed in each project.
Local evaluation	Conducted at local level		Additional information desired by local administrators		Proposed in project application   All years of grant		How does the project meet specified local needs?

Research Questions

The second national evaluation addressed four major questions listed below, each associated with a set of more specific sub-questions.

Q	1. Who is served by the program? What services do they receive? Is the program reaching the appropriate target population?

• What are the characteristics of family units participating in Even Start?

• What are the background characteristics of adults who participate in Even Start (e.g., gender, age, race/ethnicity, employment status, annual income, primary language)?

• What is the educational history of adults who participate in Even Start? What percentage of adults enter with a high school diploma or a GED and what are their characteristics?

• What are the background characteristics of children who participate in Even Start (e.g., gender, age, race/ethnicity, special needs)?

• What are the program participation histories of children who participate in Even Start?

• How do the characteristics of Even Start adults compare with the characteristics of participants in adult education programs nationally?

• How do the characteristics of Even Start children compare with the characteristics of participants in Title I and Head Start programs?

• How do background and literacy characteristics of adults and children vary across projects? Is it possible to identify sets of projects that appear to be using different targeting criteria? Do some projects appear to target "high need" populations while others target "low need" populations?

• How many families participate in Even Start? In each of Even Start's three core services? In all core services?

• How long do families participate in Even Start on a program-year basis (how many months are services received)?

Q	2. How is the federal funding spent on the program? How many of the projects are well implemented?

• What is the geographic and urban/rural distribution of Even Start projects?

• What is the federal cost for Even Start projects? What services are purchased with Even Start funds?

• How are families recruited into Even Start? How do projects target recruitment efforts?

• What local evaluation activities are being undertaken by Even Start projects?

• What is the nature of services in early childhood education, adult education, and parenting education offered by the projects? What activities do the projects offer for parents and children together? What is the nature of home visits offered by the projects? What types of support services are provided to participants?

• What types of collaborations exist between Even Start and other agencies? What types of agencies are collaborators?

• What is the training and experience of staff who provide Even Start services?

• How long do Even Start families stay in the program?

• What barriers exist to successful program implementation?

• Are the differences in program implementation associated with such factors as project cohort, length of time the project has had a grant, and area of the country?

Q	3. How well does the Even Start basic model work? Do participants perform better on key measures than similar persons who do not participate?

• What gains are observed for Even Start children on measures of school readiness and vocabulary?
• What gains are observed for Even Start adults on measures of functional literacy, GED attainment, employment status, annual income, parent expectations, and parenting skills?
• With what degree of confidence can observed gains be attributed to participation in Even Start?

Q	4. What are effective practices and programs?

• What participant characteristics (e.g., educational level of adults at entry, age of parent, size of family) are associated with program outcomes?

• What project characteristics (e.g., urban/rural location, degree of implementation, year-round versus school-year operation schedule, high federal cost per family versus low federal cost per family) are associated with program outcomes?

• What staff characteristics are associated with program outcomes?

• What service characteristics (e.g., amount of core service received, percentage of core services delivered at home versus in a center, percentage of core services delivered to parents and children together) are associated with program outcomes?

• Is it possible to define subgroups of families based on need criteria (e.g., income, entry education level), and determine whether outcomes vary across the subgroups?

While the above list includes all research questions articulated at the initiation of the current evaluation, some questions necessitated accumulating data over time. Thus, the 1993-94 Interim Report largely was restricted to descriptions of the projects and how they implemented the Even Start model. The 1994-95 report updated the descriptions of Even Start participants and their participation outcomes for the first time since program year 1992-93. The 1995-96 report added an examination of project characteristics with reference to the number of years the projects had operated Even Start.

The Even Start Information System (ESIS)

The ESIS was a modification of the NEIS used in the first evaluation. Like the NEIS, the ESIS was used to collect a common set of data from all Even Start projects to generate ongoing information about such issues as: types of projects being funded; nature and intensity of services offered; interagency collaboration; major difficulties in program implementation; participating families; and families' participation outcomes. Each Even Start grantee was responsible for completing four ESIS forms.¹⁶

Combined across all projects, the ESIS data were used to describe the Even Start program as a whole. The data also were used to categorize projects into different subgroups for further analysis. For instance, this report includes descriptions of service delivery and participation outcomes for projects grouped by their years of experience in operating Even Start and percentages of teen parents they serve.

Sample Study

Information about program outcomes for children and parents, as well as detailed participation data, was submitted to the evaluation contractor by a sample of fifty-seven projects. The Sample Study projects were selected systematically to represent urban and rural areas in all regions of the country. These projects were initially funded in 1992 or 1993. By the 1996-97 program year, some of these projects were in their fourth or fifth year of operation; a few projects had completed their grants and were no longer funded.

In addition to the same project-level and participant-level data that all the Even Start projects submitted each program year, the Sample Study projects agreed to collect more comprehensive data on participants for up to three years.¹⁷ These projects were asked to administer assessment measures to monitor the impacts of the Even Start educational services on participating parents and children. During the instrument selection process, instruments were chosen that addressed the following criteria: ease of administration for project staff; minimal burden for project participants; and appropriateness of the measure for assessing the three core areas of Even Start services (e.g., English language literacy skills of adults, the educational environment provided by parents for their children; and the effect of early childhood education on school readiness and language development for children).

The outcome measures for each of these areas is discussed further below. The specific outcomes for children participating in Even Start included the following assessments:

• The Preschool Inventory (PSI) for school readiness skills; and

• The Preschool Language Scale-3 (PLS-3) to measure language development.

To examine the outcomes of adult education, projects in the Sample Study had the choice of administering one of two tests to each participating adult:

• The Comprehensive Adult Student Assessment System (CASAS), an adult-oriented functional assessment system that measures a broad range of adult literacy skills and their application in real life domains; or

• The Tests of Adult Basic Education (TABE) Reading and Mathematics subtests for participants in GED preparation programs and some adult secondary education programs.

To assess the outcomes of parenting education, the Sample Study projects used:

• The Home Observation for Measurement of the Environment (HOME) Screening Questionnaire.

To obtain a true baseline measure of adult and child skills prior to participation in the program, only families new to Even Start were included in the Sample Study. In addition, only families in which at least one adult and one child spoke either Spanish or English were to be tested because the measures are available in only these two languages.¹⁸

Multiple parents and children from the same family could be included in the Sample Study. Unlike the first evaluation, where projects were instructed to select a target adult and child for testing, projects in the Sample Study were told to include parents who were eligible for and expected to participate in Even Start core services. Also to be tested were all children in a family who were between 2 and 5 years of age at program entry and who would be participating in early childhood education provided by Even Start or a collaborating agency.

The Sample Study projects were selected in the spring-summer of 1994 and received training on the Sample Study measures and procedures in August and September 1994. Pretest measures were administered to parents and children within thirty days of the start of program services for families who entered during that program year, and the posttest was administered at the end of the program year (1994-95), or when the family was planning to exit the program (whichever came first, as long as it was at least three months after the pretest). Families who continued into the next program year (1995-96) were posttested again in the spring of 1996. Families who enrolled in 1995-96 were pretested in the fall of 1995, posttested in the spring of 1996 (or at exit), and posttested again the following spring. Generally, there were between six and seven months between pretest and posttest.

Site Visits to Migrant Education Even Start Projects and Tribal Even Start Projects

The second national evaluation included site visits to three of the fourteen Migrant Education projects and three of the nine tribal Even Start projects operating in 1994-95. The visits focused on the special features of these Even Start projects (e.g., accommodating the seasonal participation schedules of migrant agricultural families; the geographical remoteness of tribal family homes and social/cultural factors that may influence the educational and parenting experiences of families in these set-aside funded projects). The site visits also provided an opportunity to assess the appropriateness of the ESIS for collecting data from these sites. Information gathered from the two sets of site visits has been summarized in two reports.

¹⁸  Although the child outcome measures, the Preschool Inventory and the Preschool Language Scale, were administrered in English and Spanish, the outcome measures for adults were administered only in English. The decision to use assessment insturments that measure adults' progress in English language skills reflects the programmatic goals of developing English language proficiency. Over the past sevral years, however, the proportion of the Limited English Proficent (LEP) participant population has increased, which means that the outcome measures were not equally available for all Sample Study particpants, and further, that inferences about the program are applicable only to those particpants for whom we have outcome data.

-###-

[Part 2: The First Even Start Evaluation]

[Part 4: Scope of The Second National Evaluation]