A r c h i v e d  I n f o r m a t i o n

Chapter 2

Scope of the Second National Evaluation

Universe Study

The scope of the Universe Study grew substantially during the four years of the second evaluation. The number of projects participating in the evaluation increased from 439 to 605 (Exhibit 2.2); the number of families included in the data analysis increased from 25,200 in 1994-95 to 31,200 in 1996-97.¹⁹ The tribal and MEES projects were represented in each annual data collection. While recognizing that these set-aside projects may have unique characteristics that distinguish them from other projects, in order to describe the full range of Even Start projects, we have included participant and project data from the set-aside projects in the analyses described in this final report.²⁰

In 1996-97, there were 637 Even Start projects nationwide. Of these, 95 percent (605 projects) returned at least some evaluation data by the final due date. (The distribution by state of projects that reported data for the 1996-97 evaluation is presented in Appendix B, Exhibit B.1.) Thirty-two projects that did not submit data were distributed across seventeen states in all regions. The exclusion of these projects from analyses should not bias the national program description. Similarly, in all years of the second evaluation, the analysis database included 90 to 98 percent of the projects operating in each respective year. Thus, the evaluation results presented in this report essentially represent the entire Even Start program.

Program Year and Type of Project	Number (Percent) of Projects Reporting	Total Number of Projects Operating
1993-94 Total	439 (90%)	490
State-administered Even Start Projects	420 (92%)	459
Migrant Education Even Start Projects	10 (45%)	22
Tribal Even Start Projects	9 (100%)	9
1994-95 Total	476 (93%)	513
State-administered Even Start Projects	458 (93%)	490
Migrant Education Even Start Projects	11 (79%)	14
Tribal Even Start Projects	7 (78%)	9
1995-96 Total	563 (98%)	576
State-administered Even Start Projects	546 (98%)	558
Migrant Education Even Start Projects	9 (100%)	9
Tribal Even Start Projects	8 (89%)	9
1996-97 Total	605 (95%)	637
State-administered Even Start Projects	578 (95%)	608
Migrant Education Even Start Projects	17 (94%)	18
Tribal Even Start Projects	10 (91%)	11

The evaluation findings reported in Chapters 3 through 6 are based upon data representing virtually all Even Start projects and participants?the universe of the Even Start program. The 1996-97 data analysis results provide a framework for presenting relevant, comparative information reported in previous Even Start national evaluation reports to discuss the consistencies and changes in the program implementation and participants.

For the Universe Study, descriptive statistical analyses (including cross tabulations, means, standard deviations, and univariate and multivariate analyses of variance) were used to compute (1) the numbers and percentages of projects and participants who provided specific categories of responses, and (2) averages and ranges of information such as the number of instructional hours for either all or various subgroups of respondents.

To examine participant and project characteristics that are related to participation patterns, two types of analyses were performed: (1) multiple regression analyses to identify participant and project measures that were related to various participation measures holding other factors constant (i.e., statistically excluding the effects of other factors), and (2) analysis of variance to examine further the relatively stronger relationships revealed in the regressions.

Since the statistical analyses reported in Chapters 3 through 6 were based on the universe of the Even Start program and large numbers of records, even a relatively small difference or change in percentages was statistically significant. The selection of results described throughout this report is based upon more than statistical significance alone. We discuss differences that are numerically large (e.g., 50 percent versus 80 percent) and results that appear to have practical and/or programmatic implications.

The importance of specific findings depends on the interests of stakeholders as well as statistical rules. For example, a 5-percentage-point increase in the percentage of families participating in all core services would have accounted for roughly 1,600 more families receiving full Even Start services nationally in 1996-97. This rate of increase for a typical local project with sixty families would have affected two or three families. Thus, the practical implication of this finding may be different for federal policymakers, local service providers, and the individual participants who contributed to the increase.

Sample Study

Over the course of the Sample Study, we have received data from up to fifty-seven projects, although in 1996-97, only forty-seven projects submitted valid outcome data. While this represents a decrease from the number of projects that submitted outcome data in 1995-96, it reflects the fact that some projects were not refunded and that some projects had fulfilled their obligations to the Sample Study.²¹ Exhibit 2.3 shows the number of parents and children with valid test data from the Sample Study. Individuals with out-of-range scores or missing data were excluded from these counts. In addition, posttests that were administered less than three months after pretests for the Preschool Inventory and less than two months after all other pretests were not included in the analyses because two months was deemed the minimum amount of potential instructional exposure an individual would have to have experienced in order to demonstrate any meaningful progress on outcome measures.

Measure	Number with Pretest	Number with Pretest & Posttest 1	Number with Pretest, Posttest 1, and Posttest 2 *
CASAS reading	246	117	32
CASAS math	215	115	24
TABE reading	685	286	18
TABE math	661	265	17
PSI	1,020	662	171
PLS-3 Auditory	1,023	712	188
PLS-3 Expressive	1,019	714	187
PLS-3 Total	1,016	712	187
HOME Screening Questionnaire	1,086	515	185

The number of children with valid PSI and PLS-3 pretest scores was approximately 1,100. There were fewer parents than there were children in the Sample Study (approximately 250 with pretests for the CASAS and 675 with pretests for TABE). The number of adults was lower for two reasons: (1) the TABE and CASAS were administered in English only; and (2) there were more likely to be multiple children but only one parent tested per family. These parents and children together represented 282 families with at least one test score at pretest and posttest for both adults and children.

The number of parents and children in the Sample Study was lower than expected. This is due to the following enrollment and participation patterns. Many projects were in their second or third year of operation when the Sample Study began, and they did not enroll many new families in the fall of 1994 but continued to serve families who had joined in previous years. Therefore, to augment the number of families with test data, we asked projects in the Sample Study to include those new families who enrolled in Even Start for the first time during the 1995-96 program year. It is clear that larger numbers of families enroll in Even Start than the number of pretest and posttest scores would indicate. (In other words, far fewer families have scores for the pretest and one or two posttests than have scores for the pretest alone.)

The number of adults with data from both the pretest and one posttest (who can be part of an analysis of change or gain) dropped by nearly half of the original number at pretest; for children, the reduction in sample size was less dramatic. We want to emphasize that the number of test points serves as a rough proxy for participation in the program; families who participated in Even Start for longer periods of time were more likely to have participated in additional rounds of testing. The families with only pretest data differ somewhat from those with pretest and posttest data on demographic variables such as family income, employment status, and language spoken in the home. Families with both pretest and posttest data?in other words, families who have participated for longer?on average, were more likely to be employed, to have higher incomes, and to speak languages other than English at home. Significantly fewer families with pretest and posttest data were headed by a single parent (41 percent) than families with only pretest data (50 percent) (c 2=19.59, p<.002). In addition, mothers? education among the group with pretest and posttest data was almost a grade-level higher than among the pretest only group (9.2 years versus 8.6 years).

We hypothesized in earlier reports that the group with both pretest and posttest data may have more supports within the family because there is a spouse or partner to share responsibilities. This hypothesis continued to be supported by data collected from exit interviews. There were also modest differences in reasons for exiting the program between families with pretest only and families with both pretest and posttest data. Families with both pretest and posttest data were more likely to leave because they had completed their goals (22 percent, contrasted to 18 percent for families with only pretest data, p<.01). Further, families with pretest only test data were also more likely to leave for such reasons as lack of interest, poor participation, or other problems (29 percent, compared to 23 percent for families with additional test information, p<.01). The group of families with both pretest and posttest data (again, the families who have remained in the program for longer) may not be representative of all families in Even Start. Because of these differences, we believe the Sample Study data may not reflect the experiences of other Even Start families. This key caveat should be kept in mind when interpreting the program effects presented in Chapters 7 and 8 of this report.

²¹  These numbers include all projucts that submitted any data as part of the Sample Study. As a result, the analyses described in Chapters 7 and 8 are based upon data from between forty-seven and fifty-three Sample Study projects.

-###-

[Part 3: Design of the Second Even Start National Evaluation]

[Chapter 3: What Are The Needs Of Even Start Familes?]