![[Table of Contents]](/icons/up.gif){kind=icon}
In this section we discuss the direction of Even Start research. This section discusses both improvements to existing research efforts, and ways in which external factors (e.g., welfare reform) are influencing the program, and by extension, research on Even Start.
Congress recently enacted important amendments to the Even Start program, as part of the Department of Education's appropriations act for FY 1999. This legislation stresses the need for local evaluations to collect data on program effectiveness, and requires the Department to provide technical assistance to states and Even Start projects to ensure that local evaluations provide accurate information on the effectiveness of local projects. The legislation also requires states to develop results-based indicators of program quality, and to use these indicators to monitor, evaluate, and improve Even Start programs.
The first step toward a better understanding of the quality of Even Start programs was taken by the Department in the mid-1990s when it funded the development of a Guide to Quality for Even Start Programs (Dwyer, undated). Intended as a vehicle for helping program staff design their programs rather than as a set of performance standards, the guide specifies quality considerations for each of 10 programmatic areas including integration of components, collaboration, recruitment, parenting education, home visiting, adult education, early childhood education, retention of families, staff development, and transitions. Research has linked some of these considerations to child and adult outcomes, while others are based on best judgments. The guide has been widely disseminated and is helpful to State Coordinators and project directors because it offers a guide for self-review and improvement.
The national evaluation and local Even Start evaluations currently do little to measure the quality of program services. But quality issues are especially important for Even Start because projects are mandated to use local collaborative arrangements, when possible, as the vehicle for providing services. These services, as well as those provided directly by Even Start, must be of high quality if Even Start is to produce the literacy outcomes expected of the program.
Data on the quality of Even Start projects could be collected either by moving towards a program self-assessment of quality based on "inputs" (as Head Start currently does) or by defining and measuring Even Start quality in terms of participant outcomes. Because of the key role that collaborating agencies play in many Even Start projects, it would be particularly helpful to learn whether Even Start projects choose collaborators in a systematic way that ensures quality, or whether projects are driven by the requirement to build on existing services to the detriment of quality. A study that examines Even Start's collaborative relationships could address questions on the extent to which collaborators providing instructional service share the same outcome goals, have a focus on literacy skills, see themselves as part of the Even Start program, and share data on families with Even Start core staff.
Quality Inputs. We can define high-quality program services as those that have characteristics which, in previous research, have been shown to be related to positive program outcomes. According to this definition, we know a fair amount about what a high-quality early childhood education program looks like (in terms of group size, teacher certification, and developmentally-appropriate activities). However, we know much less about how to define a high-quality adult education program(12) or a high-quality parenting education program. We also can consider the quality of other aspects of Even Start, for example, administration, coordination, family advocacy, and support services. Two sources that may help us differentiate high versus low quality in these areas are the Guide to Quality in Even Start (Dwyer, undated) and the Head Start Performance Standards (Head Start Bureau, 1992).(13)
Quality Outputs. Another way of assessing program quality is to measure program outputs. By this definition, a program that produces good program outputs is a high-quality program. Congress recently enacted legislation requiring results-based educational performance measures for children participating in Head Start programs. Similarly, Even Start has established preliminary output standards through the Department of Education's leadership in complying with GPRA and in its performance indicator plans (U.S. Department of Education, 1997). And, the recent Reading Excellence Act specifies that states must develop indicators of quality outputs, both for adults and children, for use in monitoring, evaluating, and improving programs.
Given this interest in measuring quality outputs, it would be possible to set performance goals for individual Even Start projects as well as for the program as a whole. Such performance goals might, for example, take the following form:
Setting standards assumes that there will be a measurement system for assessing whether those standards are met. In the ESPIRS, Even Start already has the basis for such an assessment system, and the Department's Even Start performance indicator plan (for use in complying with GPRA) represents a first step in setting output targets, if not standards for the national program.
There are at least two ways in which Even Start, and family literacy more generally, can be seen as becoming institutionalized: (1) the statute limits Even Start projects to a maximum of eight years of federal funding, but projects can obtain other funding to continue operations; (2) the influence of Even Start can be broadened at the federal and state levels by including family literacy services as components of other federal programs or by providing state funding for Even Start-type programs.
Continued Local Funding. After a maximum of eight years of federal funding, an Even Start project no longer is eligible to receive federal Even Start funds. While not a central or explicit goal of the program, a reasonable indicator of Even Start's success is the extent to which projects continue providing family literacy services after their federal grant has expired. The issue of project self-sufficiency is being studied by Ann Martinez at Texas A&M University who has sent out surveys to each of the original 1989 cohort of projects to find out what has happened after they completed their eight year federal funding cycle. This study will examine whether these projects were able to continue family literacy services after Even Start, and if so, how they did it.
Expanded Federal/State Funding for Family Literacy. Another possible outcome for Even Start is the generalized institutionalization of family literacy programs in communities. There is substantial evidence that this is happening at the federal level. For example, recent legislation including the Reading Excellence Act, the Adult Education and Family Literacy Act, and the Head Start legislation, greatly broadened the scope of federal programs that include family literacy services.
There is a similar proliferation of family literacy services and programs at the state-level. Several states, including Kentucky, Arizona, South Carolina, and New Mexico, have introduced or passed their own family literacy legislation. In 1989, South Carolina provided $2.3 million in funds for parent education and family literacy pilots in 21 school districts. In 1993, legislation required that all South Carolina districts implement similar programs based on the experience of the original districts. Kentucky has a state-funded family literacy program, the Parent and Child Education program.
These federal and state efforts are important evidence of the increasing appeal of family literacy and could prove important to the self-sufficiency of Even Start projects after the conclusion of federal Even Start funding.
Each year about 40 to 50 percent of the families in Even Start rely on government assistance for their primary source of income. Thus, changes in the federal welfare system have important implications for Even Start participants. In particular, the Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) brought significant changes in time limits and work requirements for welfare recipients. The federal law now limits lifetime welfare benefits to five years, with the provision that 20 percent of adult recipients can be exempted as hardship cases. The law also requires an increasing number of adults to work each year, with half of the recipients having to work at least 30 hours a week by 2002. Through the waiver system, states have been allowed to set more rigid limits in terms of the benefits that are allowable and the time frame in which adults and teens can receive these benefits.
The implementation of the Temporary Assistance for Needy Families (TANF) block grant to states under the PRWORA has affected Even Start projects in a number of ways. Program staff are adapting their adult education, early childhood education, parenting, and parent-child interaction services to meet the needs of welfare recipients.
Adult Education. The major assumption that has guided the delivery of adult education services in Even Start is that basic skills education is critical in enabling adults to carry out their responsibilities as economic providers, parents, and teachers of their children. Welfare reform has shifted the types of Even Start services for adults away from non-work-related adult basic education activities and towards job preparation and job search skills taught in a literacy context and support services that focus on job placement services. Depending on the state's laws, Even Start staff work with local welfare agency case managers to design services that qualify under the welfare reform law and that include a basic skills component (usually GED preparation or ESL instruction). Often, the time allotted to basic skills has to be reduced to allow for the inclusion of work-readiness activities taught in a literacy context.
One strategy that some Even Start staff are using to address the reduction in time for direct basic skills instruction is to teach basic skills in the context of job preparation skills. For example, adults are taught reading and writing in the context of preparing a resume and completing job application forms and letters. Participants also are taught the higher order skills of problem-solving and decision-making in their work preparation experiences. In job shadowing, participants observe a variety of jobs, analyze the skills required for these jobs, and then compare the required skills to those that they possess or are working to improve.
To meet the welfare reform work experience requirement mandated in many states, Even Start staff in projects where services are in one location, such as an elementary school, are working with building personnel to arrange for work experience placements in the cafeteria, housekeeping, or the office. In this type of arrangement, the participant is able to meet the welfare reform work experience requirement while remaining in the Even Start program setting. In states where recipients are limited in the time in which they can prepare for work, Even Start staff often offer basic skills services after the recipients' work day. This type of extended learning requires that family literacy participants commit to attending activities to enhance their basic skills after a full work day, which is difficult in terms of participants' energy and time for carrying out family responsibilities.
New welfare requirements mean that Even Start staff serving welfare recipients must develop new ways of teaching basic skills in the contexts of job preparation and job placement activities as well as offering extended learning experiences for working participants. These requirements also have led to new outcomes for which the program may be held accountable. While achievement of gains in basic skills has been the traditional outcome for the adult education component of Even Start, job placement and job retention now are being required for welfare recipients participating in family literacy programs. This change in program goals has prompted Even Start staff to be concerned not only about the participants' learning but also about the status of these individuals with regard to work readiness and work attainment.
Early Childhood Education. Even Start projects serving welfare recipients have changed the amount and the schedule of time that early childhood education is available for recipients' children as well as the content of early childhood instruction. In programs where the schedule has been extended to accommodate adults' participation in work experience as well as classes, the time for early childhood education also has been lengthened. In some instances, this has changed the demand for support services such as transportation, where additional services are needed. Projects that offer extended basic instruction for working participants are providing activities for children while their parent is receiving instruction in basic skills or ESL.
One way that Even Start projects integrate content between early childhood education and adult education is by incorporating work themes in early childhood instruction. For example, an early childhood education instructor may address the same occupational content that is being taught in adult education by using children's books about these occupations and having the children dress in costumes representing the roles of workers in the occupations. The intent is to socialize children to the world of work and to provide content that can be used in discussions between parents and children. This type of integration requires close coordination between instructors and the resources to purchase additional materials for the early childhood instruction.
Parenting Education and Parent/Child Time. With the change in schedule of adult education and the requirement for work, some Even Start projects have had to reduce the frequency with which parenting education is offered. As Even Start adults adjust to the demands of welfare reform, demands which place new time stresses on them, project staff often reframe parenting instruction to include issues that extend beyond the content usually addressed in parenting education. For example, some Even Start projects include time and fiscal management as part of parenting education.
All Even Start projects are required to provide some services to parents and children in a joint setting, often with the intention of improving the child's cognitive skills. The time changes necessitated by welfare reform often result in a reduction in the amount of parent and child time or in a change in the way this component is delivered in Even Start projects. Some projects now provide parents with packets of materials that they can use with the child in carrying out exercises at home. Parents are instructed how to use the materials prior to taking them home and then discuss their activities afterwards. In projects with a substantial home visit component, the home visitor may observe the parent-child interaction in the home and provide guidance to the parent in these activities.
Implications for Evaluation and Program Effectiveness. The adaptations that Even Start projects are making to address welfare reform have implications for evaluation and program effectiveness. The national evaluation's ESPIRS system needs to reflect the new types of activities that are being offered in each core service area, particularly in adult education concerning the provision of extended basic skills instruction and career awareness, job search, job preparation, job placement, and job retention/follow-up activities. Since the time in which activities are offered has changed, an accurate accounting of the time that participants spend in each core service area continues to be important. In addition to changes in program components, Even Start projects have extended their partnerships and collaborations to include new organizations and agencies, and these new arrangements need to be documented. The types of outcomes for Even Start participants and the measurement of these outcomes may well change as a result of welfare reform.
Perhaps most important, the changes described above including a reduction in the amount of time for adult basic skills instruction, a refocusing of parenting education away from literacy and towards life skills, and a reduction in the amount of parent/child time offered by projects, run counter to the theory underlying Even Start. Some of the positive findings from past evaluations, summarized earlier in this report, were that larger amounts of parenting education were associated with higher PPVT scores for children, and that larger amounts of parent/child time were associated with better home literacy environments. If complying with welfare reform means that Even Start has to back away from its emphasis on parenting and parent/child activities, the program could become less effective in some areas.
Alternatively, other outcomes such as increased levels and rates of employment and reduced welfare dependency may assume a more important position in Even Start. The expectations for participants and the measures used in documenting outcomes need to be reviewed to determine their appropriateness in light of program changes.
Since Even Start's first year, legislation has included evaluation requirements at both the local and national levels. These two levels of evaluation respond to differing information needs of the Department and local Even Start projects. Though the legislative mandate has changed slightly over the years, the national evaluation's basic purposes have remained the same - to describe Even Start projects and participants, to examine the performance and effectiveness of Even Start projects, and to identify effective Even Start projects for use in program improvement and technical assistance. Two cycles of four-year national studies have been completed, and a third national evaluation is underway. There is substantial continuity across the three national evaluations, but each had its own special focus and challenges.
First National Evaluation (1989-90 through 1992-93). The first national evaluation was broad in scope, addressing questions such as: "What are the characteristics of Even Start participants? How are Even Start projects implemented and what services do they provide? What Even Start services are received by participating families? and What are the effects of Even Start on participating families?" To answer these questions the evaluation developed the National Evaluation Information System (NEIS) which used paper and pencil forms that had to be key-entered as well as optically-scannable forms to collect data on participant characteristics, project implementation, and participant outcomes from all projects. In addition, the evaluation included an experimental component (the In-Depth Study) in which families in five sites were randomly assigned to be in Even Start or a control group, and were measured three times over an 18-month period.
The study provided useful information about Even Start's early implementation but it did not provide solid answers to questions about Even Start's effectiveness. Data on literacy outcomes were collected on children and adults in all projects. These national data showed consistent gains over time, but without a control group there was no way to know whether the gains resulted from participation in Even Start. The In-Depth Study provided experimental evidence about Even Start's effectiveness, but it suffered from a small sample size (only five projects were able to commit to the experimental design), sample attrition from that small original sample, and a lack of information on the control group's experiences.
The first national evaluation did describe whether early Even Start projects were able to implement the program as intended. Careful documentation of program implementation helped the Department and grantees agree on the definition of key program terms by answering questions such as "What counts as adult education in Even Start?" and "Who counts as a program participant?" The Department used data from the first evaluation to identify areas where Even Start projects needed technical assistance; in particular, improving the literacy focus and intensity of their parenting education components, engaging adults in adult education, and recruiting and retaining families.
Finally, information from the first national evaluation was used to improve the program through legislative changes. Findings that showed a fairly low year-to-year retention rate were used to modify the legislation to require year-round services so that families would remain involved in Even Start throughout the summer and into the next year. Other substantive changes made to the legislation that were informed by the study included focusing program targeting on those most in need, requiring that projects serve at least a three-year age range of children, allowing projects to serve teen parents, and allowing the involvement of ineligible family members in appropriate family literacy activities. The finding that there was a relationship between the amount of participation in Even Start and child/adult test gains, coupled with similar early findings from the second evaluation, provided evidence that resulted in an amendment in 1996 requiring Even Start services to be intensive.
Second National Evaluation (1993-94 through 1996-97). In the Department's second national evaluation the data collection instrument was improved, converted to a PC-based system, and renamed the Even Start Information System (ESIS). The administration of literacy tests was restricted to children and adults from a 10 percent sample of projects (approximately 60 out of 600), called the Sample Study, although program and participation information was collected from all projects. No control group component was included in the second evaluation.
The Sample Study was intended to provide a national snapshot of Even Start's outcomes over a four year period, and to relate Even Start practices to outcomes for use in designing technical assistance to projects. However, small within-project sample sizes, project and family attrition from the sample, and questionable quality of test data collected by local project staff left the Sample Study unable to provide information on the outcomes of individual Even Start projects or on effective practices, although it was possible to aggregate data and examine Even Start's outcomes across all projects in the Sample Study. Further, without a control group, the Sample Study could not address lingering questions about Even Start's impact.
While the second national evaluation had flaws, it provided the Department with useful information to improve the program nationally. Implementation data collected through the ESIS allowed the Department to track changes in the population served over an eight-year span, and the Sample Study provided national-level data on the size of gains made by Even Start participants. Early ESIS data provided evidence corroborating the positive relationship between service intensity and family outcomes found in the first national evaluation.
Perhaps the most important innovation in the second national evaluation was a computer program that helps grantees use national data at the local level by generating summaries of their own ESIS data. Another innovation was the development of annual profile reports for each Even Start project that could be used for continuous program improvement. First sent to each project and State Coordinator in the fall of 1997, these reports compare each project's data on several important variables to state and national data, and to other projects with similar characteristics.
Third National Evaluation (1997-98 through 2000-01). The Department funded the third national Even Start evaluation in the fall of 1997. The data collection instrument was again updated and renamed the Even Start Performance Information Reporting System (ESPIRS). Major improvements to the reporting system include the addition of a section asking parents to report the types of literacy-related activities and behaviors in which they and their children engage as well as the kinds of literacy-related tasks that their children can perform, and an updating of the project profile system developed in the second national evaluation.
To respond to criticisms that the first and second national evaluations had not adequately addressed the issue of Even Start's effectiveness, the third national evaluation provides for an experimental study to test the effectiveness of Even Start in 20 fully-implemented projects with a total of 400 Even Start families and 200 control group families.
Each of the national evaluations has shown that Even Start is a complicated program that takes time to understand and implement fully. The focus on literacy for the family as a unit is a special challenge, as is the requirement to build on existing services to create a unified family literacy program in a community. Maintaining a literacy focus and ensuring that collaborators share the goals and objectives of Even Start are critical, since Even Start projects are held accountable for literacy outcomes, regardless of collaborators' objectives. By integrating the three core instructional components, Even Start projects seek to provide a value-added dimension to families' literacy experiences and outcomes. These characteristics make Even Start unique and exciting, but also difficult to evaluate.
Working Toward Wider Use of Evaluation Data. The first two national evaluations were successful at describing Even Start participants over time. For example, the evaluation identified a dramatic rise over time in the percentage of non English-speaking families who participate in Even Start. How to best serve this growing population is one of the most important future issues for Even Start. What the national evaluation needs to do a better job of in the future is capturing the results of Even Start and ensuring that the information obtained is useful for many different stakeholders.
The key to collecting useful information is to focus on data that will improve the program. Given the need for outcome information in order to identify effective practices and the requirement of GPRA to report on program performance indicators for Even Start (as well as for all other Departmental programs), continuous improvement that is based on rigorous and objective assessment will be important to the future of Even Start. The Department will continue to examine ways to strengthen the outcome portion of the national evaluation, with one option being to return to the practice used in the first national evaluation, in which outcome information was collected on all families.
Working Toward High Quality Data in the National Evaluation. The national evaluation has been made possible by a partnership between the Department, national evaluation contractors, and local projects. Except for the first evaluation's In-Depth Study and the planned Experimental Study in the third evaluation, the national evaluation has relied on local projects for data collection. This means that data quality has varied across projects and across evaluation years, at times to the detriment of data usefulness at any level. For example, outcome data from the two most recent program years (1995-96 and 1996-97) no longer show a positive link between the amount of participation and child or adult literacy outcomes. It is possible that this results from a lack of consistency across projects in the administration of child and adult literacy tests.
As the Even Start program grows larger (from fewer than 100 grantees in 1989 to more than 700 in 1998), data quality and the resources it takes to train all Even Start grantees to collect data grow in importance. While it would be possible to estimate national statistics based on a sample of projects, there are several good reasons to include all Even Start projects in the national evaluation. First, the more projects involved, the better able the evaluation will be to identify promising practices, through correlational analyses and case studies. In addition, the national evaluation is currently the source for all tracking information on projects. Without a universe data collection, the Department would have no comprehensive list of all Even Start grantees or their characteristics. Finally, having all local projects collect a common set of data means that analyses can be done at the national, state, and local levels.
Balancing National and Local Evaluation Needs. Another challenge in evaluating Even Start is balancing the data needs of national and local evaluations. In response to legislative mandate, the national evaluation is meant to track trends over time, provide performance information, identify promising practices, provide a national and state comparison for local projects, help the national program office identify technical assistance priorities, and improve Even Start nationally. At the same time, local projects are required to conduct a local evaluation to help improve services and demonstrate positive results to appropriate stakeholders.
While recognizing that the national evaluation is evolving into a more useful tool for local projects, project staff worry that they are asked to spend time and energy collecting and submitting data for the national evaluation, but that those data are of limited use at the local level. Project staff have commented that the national evaluation could do a better job of capturing what goes on in individual Even Start projects, thereby illustrating the diversity of projects across the country. Unfortunately, the characteristics of the universal data collection instrument which make it useful for informing Congress about the national Even Start program also make it less useful to individual projects. The Department has stressed that the national evaluation must first respond to national information needs, which means, by definition, that it cannot provide data which are maximally useful to local projects. Though the national evaluation has been limited in its ability to serve local needs, progress is being made.
The Department, through the national evaluation and other activities, has tried to:
The State Role in Evaluation. States administer the Even Start program and need some sort of accountability tool. Yet at present, there are no standards for states to use in deciding whether their Even Start projects are making sufficient progress. Given this information gap, it is not surprising that several states have assumed responsibility for planning and conducting studies of their local projects, either by using administrative funds or by coordinating local project evaluations.
The existing state role in evaluation seems poised for change because the recently-enacted Reading Excellence Act amended the Even Start law by requiring states to develop results-based indicators of program quality to be used in monitoring the performance of local projects. States now have a unique opportunity to set results-based quality standards, and having state standards could improve evaluation at the local level and increase accountability. At the same time, the new prominence of states in Even Start evaluation could lead to overlap among national, state, and local evaluation activities. Careful coordination among these different levels of evaluation will be needed to avoid overburdening local projects.
Complementary Evaluation Studies. Because the national evaluation cannot answer all of the research questions that are asked about Even Start, the Department has funded additional studies of Even Start, each with a special focus. For example, the Observational Study of Even Start Projects is studying 12 projects that are at least in their third year, that have evidence of being fully implemented, and that have produced positive outcomes for at least two years. The study is examining outcomes for families, quality of implementation, and linkages across Even Start's three components. Through this study, the Department hopes to facilitate and assess approaches to continuous program improvement in Even Start projects based on clear outcome goals for children and families, program quality standards, rigorous and objective assessment of program results, and the use of evaluation results to monitor progress and enhance program quality. As described earlier, the second part of this study will help projects improve their parenting education component. Information gathered from working with these projects over three years will help the Department provide assistance to all Even Start projects on local evaluation and parenting education.
Other complementary studies being funded by the Department include a synthesis of state and local Even Start evaluations, a synthesis of research on family literacy, a study of the continuity of services between Even Start and Title I, and an analysis of the population eligible for Even Start. All of these should provide useful information in helping to frame the future Even Start research agenda.
Even Start has an ongoing national evaluation for all grantees that informs Congress and the Department. Future options for evaluation ought to focus on issues that will help local projects improve their services, as well as on the overall effectiveness of Even Start.
Focus on Literacy Outcomes for All Projects. Data on the literacy skills of Even Start participants is important at all levels. Locally, Even Start projects need to assess literacy progress to know what, if any, changes to make in their services. Data on literacy outcomes also are important for Even Start State Coordinators, who now must specify quality standards and decide which projects are making sufficient progress to receive continued funding. Nationally, Congress holds Even Start accountable for progress toward outcome achievement targets set in the national performance indicator plan, required by GPRA. Since the national evaluation collects information on all projects, and all projects would benefit from having outcome data, one option is for the national evaluation to designate a common set of rigorous and objective outcome measures to be used in all projects, and also give local projects flexibility by allowing them to choose from a set of approved measures.
Strengthen Local Evaluation for Continuous Improvement. How can local projects best be improved? A recent analysis of local Even Start evaluations (St.Pierre, Ricciuti & Creps, 1998) shows that local projects do not consistently produce high-quality, useful, local studies and calls into question the utility of local evaluations, as currently structured, for improving projects. Like many other aspects of Even Start projects, the content and quality of local evaluations varies. Variation in state and Department guidance, project maturity, available funds, and project needs contributes, not surprisingly, to variation in local evaluation designs. While the synthesis found some local evaluations that were well done and useful to projects, others were of low quality and of little use to a local project.
Program improvement efforts have been haphazard, occurring mostly through trial-and-error and through the hard-learned experiences of program staff. This is one way of improving projects, but greater gains could be made by accompanying these personal, anecdotal methods with a data-based, systematic assessment of program strengths and weaknesses. Through dissemination of the Guide to Quality in Even Start (Dwyer, undated) and through evaluation training sessions, the Department has been working with local projects to modify their approach to evaluation by engaging in such a continuous improvement effort. Relevant systematic continuous improvement efforts have been described by Haslam & Stief (1998) in an observational study of 12 Even Start projects, by Alamprese (1996) in studies of workplace literacy programs, and by Appel (1998) in her work with local Even Start evaluations.
Examine the Quality of Even Start Services. Although the Department has issued guidelines for establishing a high-quality Even Start project, there has been no systematic assessment of the quality of Even Start projects or of individual Even Start services. Options for assessing program quality include a self-assessment of the extent to which each project's services match the Even Start quality guidelines (the Guide to Quality has an accompanying program self-assessment tool) and a comparison of the gains made by Even Start families against a set of performance standards. Finally, collaborating agencies play a key role in many Even Start projects, and it would be particularly helpful to do a study of the collaborations that Even Start projects arrange. Such a study could examine whether Even Start projects choose collaborators in a systematic way that ensures quality, or whether projects are driven by the requirement to build on existing services to the detriment of quality.
Continue to Study Short- and Long-Term Effects. The basic question about Even Start's effectiveness has not been answered to everyone's satisfaction. The first national evaluation provided useful data on learning gains from all projects, as well as data from five projects which implemented high-quality randomized experimental evaluations. However, no group of five projects can represent Even Start very well, and the findings from this study, while important, cannot be generalized to Even Start as a whole. The second national evaluation provided data on learning gains from a random sample of all projects, but with no control group it is difficult to attribute gains to Even Start. The third national evaluation plans to provide better answers about Even Start's effectiveness by implementing randomized studies in 20 projects across the nation, and by collecting pretest/post-test data on child and parent behaviors from all Even Start families in all projects.
Through the studies done to date we have learned that Even Start adults and children make short-term literacy gains and that many adults acquire a GED, even though the average length of participation is less than a year. While the effect on acquisition of a GED seems clearly attributable to Even Start, we are unsure of the extent to which the literacy gains are due to Even Start or to normal development. This question will be addressed by the experimental study currently being conducted as part of the third national evaluation. If early findings are favorable, the experimental study could be extended to determine whether the short-term effects of one year of Even Start participation lead to the hypothesized longer-term effects that are illustrated in Exhibit 2.
Unfortunately, the severe need level of most Even Start families combined with the fact that most families leave the program within their first year of participation, may truncate the effectiveness of the Even Start model. Though it could not be done in an experimental fashion, it would be useful to follow families who participate in Even Start for an extended period of time, in an attempt to determine the long-term value of persistent participation.
States could help with long-term studies of Even Start. The mobility of Even Start families after they leave the program (and in many cases during their Even Start participation) complicates any national long-term follow up. However, some states are in the process of building detailed student data bases for tracking students, data bases which could be useful both at the state level and nationally to provide information on students who were once in Even Start. For example, the Florida State Department of Education has put into place a data base of student information that details student-level information each year. To examine the long-term effects of Even Start, it would be useful if data bases like these contained an element that would assign a code for each student indicating participation in Even Start. Ideally, state-level data bases would indicate any participation (current or former) in Even Start for children in all grades.
The diversity of the current set of Even Start studies, including the national evaluation, descriptive case studies, and syntheses of research and evaluations is already broadening the types of information available about Even Start implementation and outcomes. The Department has made important improvements to the third national evaluation, including the use of a control group for isolating Even Start's impact and a new set of family progress indicators. Upcoming improvements to the reporting system will focus on accurate measurement of important factors such as service intensity and participation. The control group study could also be a vehicle for examining some of Even Start's cost and quality issues. The current portfolio of Even Start studies, in conjunction with exploration of the options discussed, represents the future of Even Start's evaluation efforts.
Footnotes:
(13) Compliance with the Head Start performance standards is a requirement for receipt of federal Head Start funding. The standards specify the types of educational services, operations, and facilities that are required; the health, medical, and nutritional services that must be provided; and the social services and parent involvement that must be offered by Head Start programs.
(14) The information in this section is drawn from Alamprese & Voight (1998) who conducted five case studies of adaptations that family literacy projects have made to meet the new requirements faced by welfare reform clients. This work was done for the National Center for Family Literacy and the Knight Foundation.
-###-
[References]