[4.0 Description of Local and State Evaluations]
![[Table of Contents]](/icons/up.gif){kind=icon}
[6.0 Recommendations]
This section summarizes findings about Even Start's outcomes based on the 24 studies that were reviewed in detail. Rather than using the interpretation or conclusions reached in the individual evaluation reports, we reviewed the data/findings presented in the reports and draw our own conclusions.
Due to the great variation in questions addressed, methods used, measurements made, and approaches taken in local evaluations, the information contained in the evaluation reports we reviewed does not allow us to draw a consistent set of conclusions about the implementation or the effectiveness of Even Start projects. This is not a criticism of local evaluations. Rather, it reflects the substantial variation in the purpose and nature of local evaluations. Hence, in this section we focus on the range of outcomes described by local projects and, where possible, provide national comparisons.
Cognitive outcomes. Local evaluations showed a consistent pattern of positive gains on several different measures of child development including the PSI, PLS-3, PPVT, pre-IPT, and COR. Each of these measures was administered on a pre-post basis by three or more projects, and in almost every case, positive gains were observed. When significance testing was done, the gains generally were statistically significant.
The size of the gains is consistent with the size of the gains observed in the national Even Start evaluation and in other family literacy studies which administered these same measures. For example, annual PSI gains in the local evaluations were on the order of 5 to 11 raw score points, roughly the same as the 7.3 points per school year gained by children in the NEIS portion of the first national Even Start evaluation,[12] the 5.1 points per school year gained by children in the Sample Study portion of the second national evaluation,[13] and the 4.5 points per school year gained by children in the Even Start In-Depth Study[14]. Similarly, the annual PPVT gains in the local evaluations were on the order of 5 to 12 points, roughly the same magnitude as the 7.5 points gained by children in the NEIS portion of the first National Even Start evaluation[15] and somewhat greater than the 3.2 points per school year gained by children in the Even Start In-Depth Study[16].
Parent ratings of effects on child. Parents were asked questions about what their child learned while in Even Start, whether their child enjoyed Even Start, and whether their child was helped by Even Start in a variety of areas, e.g., more successful in school. Parents reported, among other things, that Even Start children benefitted from the program, learned to get along better with others, enjoyed reading more, and were more likely to succeed in school.
Developmental screening. Projects administered many developmental screening instruments including the Denver Developmental Screening Inventory, the Batelle, the Early Screening Inventory, the DIAL-R, the Brigance, the Quality Reading Inventory, the HOSTS survey of Developmental Tasks, the Normal Developmental Checklist, the Learning Accomplishments Profile, the Kindergarten Readiness Skills Checklist, the Iowa Tests of Basic Skills, the Psychomotor Skills Inventory, the Early Childhood Assessment, and several locally-developed measures. Even Start children were generally seen as scoring within age-appropriate norms on standardized tests or as making improvements or gains.
Teacher ratings. Preschool and elementary school teachers were asked to evaluate the school readiness behaviors and skills of children using the Teacher Assessment of Achievement, the Growing Up Strong Observable Behavior Checklist, as well as several locally-developed measures. Teacher ratings of Even Start children's behaviors, skills, and interactions typically increased as those children aged; teachers (kindergarten and later grades) were likely to rate most Even Start children as developmentally ready for school or as doing well in school.
Observations were done of parent/child reading, child verbal interactions, and language sophistication in interactions with peers. There was no change over time in parent/child reading, but observations did reveal positive changes in communication and interaction with peers.
School record abstraction of attendance and grades showed that Even Start children have higher attendance rates than the general school population; analysis of school grades was inconclusive.
This review of the data provided by local evaluations shows that children in Even Start made pre-post gains on several different measures of cognitive development, consistent with the gains observed in the national Even Start evaluations. In addition, regardless of the particular measure used, local evaluation reports invariably reach positive conclusions about the effectiveness of Even Start on participating children.
Cognitive outcomes. As was the case for child-level outcomes, local evaluations showed a consistent pattern of positive gains on several different measures of adult development including the CASAS, the IPT, and the TABE. Statistically significant gains were observed in almost all projects that administered these measures. In addition, most projects that collected data on GED status reported that substantial numbers or percentages of adults attained a GED during their participation in Even Start.
The local evaluations that we examined report annual CASAS gains ranging from -5 points to +9 points, certainly in the same range as the average 4.6 point per year gain for adults in the NEIS portion of the first national Even Start evaluation,[17] the 4.2 point per year gain for adults in the Sample Study portion of the second national Even Start evaluation[18], and the 1.6 point per year gain for adults in the Even Start In-Depth Study[19]. The CASAS has a standard deviation of 16 points, so the roughly 4 point gains that have been seen in many studies correspond to about one-quarter of a standard deviation, a small difference by social science standards.
TABE scores were generally reported by local evaluations in grade equivalent units, with gains ranging from 0.2 grades per year to more than 2.0 grades per year, depending on the project and the subtest. As is the case with other parent measures, these are in the same range as the average gains seen in the national Even Start evaluation (about 1.3 grades per year in reading and 0.9 grades per year in math).[20]
In addition to the CASAS, IPT, and TABE described above, parents were administered various tests including the Gates-MacGinitie Reading Inventory, and the Basic English Skills Test, the Language Assessment Scale. Pre-post results on these assessments were mixed, with some studies finding positive gains while others saw no growth.
Parent surveys of personal growth and attitudes toward and involvement in education. Parents were asked to report their perceptions about the effects that Even Start had, e.g., whether they learned life skills, made friends, improved math and reading skills, improved self-esteem, improved job skills, learned time management skills, liked school more, were more involved in their child's school, volunteered more in their child's school, and learned workforce readiness skills. Parents also were asked to complete rating scales to assess the effects of Even Start in areas such as social support, family stress, self-esteem, and depression. Parents almost always had very positive perceptions about Even Start's effects on their lives and the lives of their children. However, there were few, if any, significant changes over time on rating scales of social support, family stress, and depression.
Parent self-reported educational level and employment status. Many projects asked parents (or otherwise kept track) about their educational level and employment status. Most projects that reported on these outcomes did not provide data adequate to assess change over time, e.g., projects reported the number of parents who obtained a job or an educational credential but did not report the total number of participating parents, or the number who were looking for work.
Parent surveys about goals. Parents were asked about the goals that they established at entry to Even Start and whether those goals were achieved. Parents reported that they were very successful at meeting the goals that they had set at entry to Even Start.
Parenting measures. Across the 24 projects, parents were asked to complete the Home Screening Questionnaire, the Bowdoin Inventory of Parent Beliefs and Practices, and the Systematic Training for Effective Parenting scales, in addition to many different locally-developed questionnaires about knowledge of parenting practices, parenting responsibility, and reading behaviors. Parents also were asked to report their perceptions about the effects that Even Start had in areas such as reading books to their child, talking to their child about school, taking their child to the library, becoming a better parent, knowing about child development, communicating better with their child, and helping their child with homework. Local evaluations generally were unable to detect significant changes when using rating scales of parent behaviors and knowledge. However, parents almost always had very positive perceptions about Even Start's effects on parent/child interactions.
Observations of parenting behaviors were conducted during home visits and during parent and child together (PACT) time. Projects that conducted this type of data collection reported positive changes in parenting behaviors.
Teacher ratings. Teachers were asked to rate parents in areas such as parent involvement in school, communication with school, and workforce readiness skills. There were no pre-post changes in teacher ratings in these areas.
These findings from local evaluations match findings from prior research which show generalized parent satisfaction with almost any social program (McNaughton, 1994) as well as small amounts of cognitive growth, but few measurable changes over time in social-emotional development, or parenting skills as measured by rating scales.
A program "impact" is different from a program "outcome." The outcome of a program is the gain or growth or change that is observed in participants during their time in the program. For example, the statement that "children in Even Start gain "x" points on the PSI during a program year" describes a program outcome. The impact of a program is the difference between the gain observed for program participants (the outcome for people in the program) and the gain observed for nonparticipants (the outcome for people who were not in the program). For example, the statement that "children in Even Start gained "y" points more than children not in Even Start" describes a program impact.
Impacts compare the outcomes for Even Start participants with some standard, for example, the outcomes achieved by a control or comparison group, or national norms, or the gains normally achieved by Even Start participants. While each of the 24 reports that we reviewed provided information on outcomes, only half of the reports tried to gauge the size of these outcomes by comparing them to the outcomes observed in other studies. Those that did, relied on a range of comparisons.
These local evaluations made an important attempt to judge the magnitude of the gains observed for Even Start children or adults. However, none of the studies was able to estimate the "true" impact of Even Start, which would require randomly assigning incoming families to Even Start or to a control group. Given the resources available to local evaluators, it is unreasonable to expect that any local study could actually put a randomized experiment in place.
Only a scattering of local evaluations provided any information on the way in which participant characteristics are associated with outcomes. Local evaluators do not provide such breakdowns because few local evaluations have sufficiently large sample sizes to do any but the most aggregated analyses. Sample sizes in the studies we reviewed were most often in the range of 15 to 50 families. Therefore, while some of the raw score differences between the groups compared appear to be relatively large, they are not statistically significant because the sample sizes are so small, often fewer than 10 per group.
One study examined child gains separately for boys and girls, and for whites and Hispanics, with no differences between groups. A second study analyzed TABE reading, math, and language gains for adults who participated for over/under 150 hours and showed greater gains for the more active participants. A third study analyzed data separately for "new" vs. "continuing" children, younger vs. older preschool classrooms, Spanish vs. English test administration, and boys vs. girls, with no consistent differences for any comparison. Finally, a fourth study analyzed PPVT scores by age, with no differences in the gains of younger vs. older children.
13 Even Start children in the second national evaluation's Sample Study gained an average of .64 points per month on the PSI (Tao, Gamse & Tarr, 1998). School year gains are estimated to be 5.1 points in an 8 month school year (.64 points per month * 8 months = 5.1 points).
14 Even Start children in the first national evaluation's In-Depth Study gained an average of 10.2 points on the PSI over an 18 month period (St.Pierre, et al, 1995, p. 162). School year gains are estimated to be 4.5 points in an 8 month school year ((10.2 points/18 months) * 8 months).
15 Even Start children gained an average of .94 points per month on the PPVT as measured by the first national evaluation's NEIS data collection (St.Pierre, et al, 1995, p. 173). School year gains are estimated to be 7.5 points in an 8 month school year (.94 points per month * 8 months = 7.5 points).
16 Even Start children in the first national evaluation's In-Depth Study gained an average of 7.1 points on the PPVT over an 18 month period (St.Pierre, et al, 1995, p. 171). School year gains are estimated to be 3.2 points in an 8 month school year ((7.1 points/18 months) * 8 months).
17 Even Start adults gained an average of 4.6 points in a school year on the CASAS reading scale as measured by the first national evaluation's NEIS data collection (St.Pierre, et al, 1995, p. 189).
18 Even Start adults who participated in the second national evaluation's Sample Study gained an average of 4.2 points in a school year on the CASAS reading scale (Tao, Gamse & Tarr, 1998, p.145).
19 Even Start adults who participated in the In-Depth Study gained an average of 3.7 points over an 18-month period on the CASAS (St.Pierre, et al., 1995, p. 186). School year gains are estimated to be 1.6 points in an 8 month school year ((3.7 points/18 months) * 8 months).
20 Even Start adults who participated in the second national evaluation's Sample Study gained an average of 1.3 grade equivalents per year on the TABE reading scale and 0.9 grade equivalents per year on the TABE math scale (Tao, Gamse & Tarr, 1998, p. 149).
|
[4.0 Description of Local and State Evaluations] |
|
[6.0 Recommendations] |