A r c h i v e d  I n f o r m a t i o n

Early Childhood Reform in Seven Communities - October 1996

VI. Assessment of the Outcomes of Early Childhood Reforms

This chapter describes the types of evaluation strategies in use in these seven programs; summarizes evaluation findings; presents teacher and parent perceptions of outcomes; and, concludes with observations about the status of evaluation in early childhood programs.

Evaluation Strategies

As displayed in the attached table, the study sites participated in a mix of evaluation strategies to help them improve their programs and to document outcomes. Some were part of large, multi-site evaluations and sponsor-driven efforts, such as the FACE programs and Kentucky's Family Resource and Youth Service Centers. The FACE evaluation reports provide extensive site-specific infomation on implementation issues, challenges and accomplishments. The Family Resource and Youth Service Centers evaluation is based on cumulative statewide data from project management information systems and does not provide site-specific findings. The Parent Services Project also completed several evaluation studies funded by local foundations during its early years of program development.

The second type of evaluation strategy in case study sites is small, local, low-budget evaluations contracted out to university affiliates. This strategy was adopted by three sites:

- The Jersey City school district encouraged graduate students from Columbia University to assess the impact of the preschool program on children for their dissertation research. The district is also conducting a self-evaluation and collecting data on outcomes for students, parents, and teachers. Both sets of studies are not yet available.

- Guidelines from Kentucky's early childhood education program specify that local sites should include in their grant proposals an evaluation component. With the help of foundation funds, the Biggs Center commissioned two researchers from the Early Childhood Department, University of Cincinnati to assess the quality of parent-child interactions. The study is in progress and only preliminary summaries have been provided to the program.

- Using HUD funds, an assessment of the Inn Circle initiative provides information about the population served, progress in achieving parent goals, and housing and economic status upon leaving the program. In addition, staff collect Head Start data from tracking forms and external researchers analyze the information. The Head Start evaluation has both formative and outcome components. Researchers from the University of Iowa School of Social Work conduct both evaluations, which are in progress. Periodic summaries of the evaluations are sent to HACAP.

Several other project included in this study were not presently involved in a formal evaluation, although they employed a variety of internal system for tracking the progress of families and children and for reporting to program sponsors and funding sources. For example, Child Development, Inc. reports on 77 recent participants in their Teen Parent Program show that 44 have graduated or completed their GED, 15 are still enrolled in school, and 18 dropped out before completing the program.

TABLE 3

Characteristics of Evaluations in the Program Sites

Findings from Evaluation Studies

Review of interim and final evaluation products reveals a variety of findings on program outcomes:

Parent Services Project (PSP). PSP was the subject of a formal evaluation sponsored by the San Francisco and Marin Community Foundations. The evaluation, conducted by Allen Stein and Associates from 1985 to 1988, focused on the importance of parental support in reducing stress. Using an experimental design, the evaluation examined the extent to which PSP's social support of parents acted as a buffer to reduce stress among low-income parents (Stein and Haggard, 1988). The evaluation did not measure changes in parenting behaviors and child outcomes. The study involved interviews with 169 parents in 20 PSP sites and a control group of 86 similar parents involved in child care centers that were not participating in PSP. Questionnaires about life events and stress, social support, and psychological symptoms were administered to parents upon entry, after 15 months, and finally, after another 15 months, when their children had graduated from the centers. According to the evaluation:

- PSP parents showed significant decreases in stress over the short-term and in psychological symptoms over the long-term.

- The Project's supportive effects did not strengthen informal networks outside the program.

- Cultural factors influenced the reporting of stress and symptoms.

It concluded that PSP is effective in reducing symptoms of stress; that child care alone was not sufficient to reduce levels of stress and social isolation; and that there were benefits to family support activities in child care settings.

In a follow-up study of sites set up in 1989 and 1990, consultant Molly Haggard found positive effects in the skills and attitudes of staff members and "dramatic increases in parent attendance and involvement activities." In addition, Paul Harder of the URSA Institute in San Francisco examined the cost benefits of PSP, based on state services that would be expended on PSP families had preventive support not been available. He calculated net savings to the state of California of $240 for each family served by PSP.

Family and Child Education (FACE). The FACE evaluation, sponsored by the BIA and conducted by Research and Training Associates (RTA) is an ongoing study of implementation issues. Unlike the PSP evaluation which was outcome oriented, the FACE is a formative evaluation focusing on service delivery. The core of the evaluation is to obtain baseline data for long-term outcome analyses, describe program implementation efforts, and assess the evaluation design to determine its usefulness to the BIA and to make necessary changes (RTA, 1992 & 1993).

The evaluation design calls for on-site data collection in eleven sites by local staff members; it also relies on reports from trainers on the strengths and challenges of implementation, and site visit interviews and observations conducted by the evaluators. Site staff members keep recruitment and activity journals; child and parent enrollment and service information; and child observation assessments and tracking of parents' progress in obtaining a GED, high school credits, or college credits. Some of the early outcome findings from this study are:

- One-third of adults enrolled in the adult education program in 1992 received their GED or high school diploma.

- Children showed improvements on the Child Observation Record, an instrument developed by High/Scope to measure developmental progress.

The evaluation also pointed to broader, less tangible impacts of the program. Based on interviews with staff and parents, the evaluators reported:

 

- Kindergarten teachers reported that children who had gone through FACE adjusted more readily to kindergarten compared to other children.

- Parents reported improved parenting skills.

- Parents and staff reported an increase in parents' self-worth and confidence, and in parental awareness of the importance of education.

- The program's impact spilled over to the community, in terms of cultural pride, participation of community members in FACE events, and hiring of local people to serve on staff.

- Staff developed new understandings of working with families, team teaching, and their roles as "facilitators of learning."

The evaluation also concluded that the FACE program has evolved into a distinct model through blending and adapting the national models; and that intensive training, unique to this program, is essential to program effectiveness.

J. E. Biggs Early Childhood Center. Given its strong emphasis on parent involvement, it comes as no surprise that the Biggs Center chose to examine outcomes related to parent-child interaction. Data collection involved pre- and post-videotaped sessions of parent-child play sessions with a group of 28 program parents and 20 control group parents who had children in another local child care center without a parent involvement component (McCollum and Yates, 1994). Other data included checklists on child development behaviors and interviews about discipline and teaching strategies. Data are also collected on children's competencies by the time they reach kindergarten. Overall, early findings report positive outcomes of the Biggs Center's family-oriented approach to early education:

 

- Program parents showed improved interactions with their children compared to the control group.

- The best predictor of the quality of parent-child interactions was the accuracy of the parent's knowledge of her own child's development.

- Although findings were not statistically significant, more program parents showed positive parenting behaviors between pre- and post-interviews.

One useful result of the study was that the control group site has developed a three-year plan to develop parent involvement goals and activities.

Overall, these studies have found modest positive effects on different dimensions of an early childhood programs: PSP, on parent functioning; FACE on adult education completion and child development; and the Biggs Center on parent-child interactions. However, with the exception of PSP, the evaluations are still at preliminary stages and programs await the final results. Also, while programs in our study were comprehensive efforts to serve both children and parents, evaluations assessed child outcomes, parent outcomes, or the interaction of parent and child -- but rarely all three.

Sites reported using evaluation results to support dissemination and replication efforts. The PSP evaluation was instrumental in legislative bills to promote statewide adoption in California and has become a vehicle for national dissemination. It was used by the state of Delaware to secure federal funds for a substance abuse prevention program modelled on PSP. Ethel Seiderman, PSP's director, uses data about the levelling off of parent support networks after parents leave PSP to advocate continued family support activities in the school system. PSP also responds to requests for information about its program by including a summary of the evaluation findings. Similarly, the FACE and Covington evaluations promote scale-up; the FACE program offers information to support its goal of expansion to all 144-BIA funded schools and the Covington directors use evaluation data in workshops for principals and other school administrators to promote a family approach to education in elementary schools and to secure grants for district-wide parent involvement activities.

Findings from Focus Group Interviews

The atmosphere of early childhood agencies is rich with testimonial data on the impact of program services. Interviews with parents reveal impressive anecdotal support of these initiatives. Parents attested to the value of the programs in terms of promoting children's social skills and school readiness, providing parents' with knowledge on their children's development and how to work with them at home, and giving parents' the confidence and skills for their own advancement.

"PSP helped my daughter to be ready for kindergarten and helped me to be ready to be involved."

"Raising kids out in the sticks where you don't see other kids, sometimes you think they're mentally retarded. When you see other kids their age, it helps."

"When people question the value of subsidized child care, I like to tell them that I am proof that the system does work. It was tough to finish school, work during the daytime and take care of my kids. I can't count the times I turned to my Family Support Coordinator, or the number of times she read my face and knew I needed some encouragement. And when they first drew me into being involved in the parent organization I said, 'I can't do this--stand up in front of a crowd and talk?!' But now I'm a member of the Georgia Child Care Council-appointed because we bombarded the Governor's Office to add a parent representative."

Similarly, focus groups with staff members elicited their views on the outcomes of program participation. They pointed to success stories in terms of greater parent involvement, improved parent-child interactions, and helping parents achieve their goals.

"I'm seeing parents learning how to be good listeners to each other in support groups, and then translating that ability to their interactions with their children."

"Our parents started coming to the classroom. They had rarely advocated for bilingual education before, but now they tell us that they like what we do and that they want it to continue."

Testimonies from parents and direct observations of clients' successful progress following participation in early childhood programs have a powerful form of "in your face" validity. However, there are also some risks in using anecdotal evidence: focusing selectively on a top tier of families that represent dramatic "turnaround" examples of program impact; and a lack of disciplined attention to those children and families who drop out or fail to thrive within a program. Furthermore, in the case of child outcomes, it is difficult for parents and staff to separate out the contribution of program experience and the results of normal maturation processes.

Conclusions

Stepping back from the details of these evaluation strategies and findings, we offer two additional observations from the experience of these seven initiatives. First, the climate of early childhood programs is one where priority lies in direct services rather than issues of infrastructure. There is a dearth of external evaluations and not much technical assistance for developing local self-evaluation tools for program improvement. One speculative explanation for this situation lies in the substantial unmet demand for early childhood services. This fact may also play a role in the perception of local managers on issues of quality, outcomes and accountability. When programs have long waiting lists, managers feel that the value of their services is validated by consumer demand.

Secondly, there is substantial congruence between the design and implementation of these projects and the large body of research literature cited in Chapter II. As noted, the FACE program design is based on three models (Parents As Teachers, National Center for Family Literacy, High/Scope) which have been evaluated, and certified as effective by the Department of Education's Program Effectivenss Panel. Similarly, classroom practices, staffing patterns, and strategies for working with families are guided by research, as embodied in program guidelines and position statements of professional organizations. In this cumulative sense, most local early childhood programs are examples of the utilization of research and evaluation data.

Thus, although we do not have solid information about children's outcomes from these programs, past research has shown that children who participate in high quality early childhood programs benefit in social, emotional and cognitive development. The programs in our study do provide children a stimulating environment and support their families. Based on staff perceptions, children enjoy the programs and children's poor attendance has never been an issue. It is quite likely that these children will show good outcomes as they transition to kindergarten. They stand a better chance of early school success compared to children who do not participate in early childhood programs or in programs of lesser quality.
-###-

[Cross-Site Analysis - Policy Influences in Local Agencies] [Assessment of the Resources Required to Implement Reforms]