A r c h i v e d  I n f o r m a t i o n

3.0 Study Methods

Research Questions

At the start of this study, the Department posed a series of questions about what local evaluations look like and how they are conducted, as well as questions about the kinds of findings reported in local evaluations. Analyses of local evaluations are limited in the extent to which they can contribute to our knowledge about Even Start's program operations and effectiveness. This is due to several characteristics of local evaluations, such as: variation in the kinds of studies done at the local level, limitations of local evaluation designs, and issues with the quality of local evaluations. This report describes local evaluation studies in Section 4 and summarizes a limited set of the findings from those studies in Section 5.

Obtaining Evaluation Reports

During 1996-97 there were about 650 local Even Start projects, and the law specifies that each project ought to conduct a local evaluation. At the same time, some states have planned and are conducting their own evaluations. For the present study, we hoped to collect at least 100 local and state evaluation reports, though we had no idea how many reports could actually be obtained.

Local evaluations are conducted for the benefit of local projects, and those projects are not required to transmit local evaluation reports to the Department. However, some State Coordinators require their grantees to submit copies of their local evaluations. Therefore, we sent a letter to all State Coordinators, describing the planned synthesis and asking for their help in obtaining copies of state and local studies.

Obtaining local evaluations was difficult (see Exhibit 2). A few State Coordinators sent us copies of the local evaluations they had received. In other cases the State Coordinator did not have copies of local evaluation reports, but they were able to send us a list of local projects or local evaluators and suggested that we call projects directly in order to obtain evaluation reports. In 33 cases, the State Coordinator did not respond to the request.

With limited time and funds, we ended up with a sample of convenience - a total of 4 state evaluations (1 each from Hawaii and Nevada, and 2 from Oregon) and 118 local evaluations from 19 states. We do not know how well the 118 local evaluations represent all local evaluations.

Receiving and Logging-In Reports

As we received evaluation reports we created a data base by logging-in each report and recording selected pieces of information. The resulting data base allowed us to produce a complete listing of all reports received in this project and to describe the universe of state and local reports submitted to us.

The log-in form was developed on the basis of (1) a review of the evaluation literature concerned with assessing the quality of research studies, and (2) the needs of this project to capture a variety of study types. There is substantial literature on methods of assessing the quality of an evaluation (e.g., Cooper & Hedges, 1994). However, this literature focuses almost completely on the quality of evaluations which seek to determine the effectiveness of a given program, and hence is concerned with issues of random assignment, control groups, sample sizes, and so on. The same holds for the very large literature on meta-analysis. We examined samples of the coding schemes used in these studies and used variables which were most relevant to the present context.

We found little research literature on assessing the quality of implementation studies. This is unfortunate because we expected that many of the studies conducted by Even Start projects and states would focus on implementation issues. Because we did not want to neglect evaluations that focused on implementation issues, we devised our own coding scheme for implementation studies.

Describing Methods Used to Conduct Local Evaluations

We used data from the log-in form on all 122 evaluations received to describe the kinds of studies that were done at the local level. These data allowed us to calculate the percentage of studies that focused on program outcomes and to describe the design of those studies, the time period covered by the studies, the sample(s) measured, and the kinds of measurements that were made. We also calculated the percentage of studies that focused on implementation issues to describe the design of those studies, the sample(s) measured in those studies, and the kinds of measures that were used.

Describing Findings from a Subset of Evaluations

We selected a subset of 24 local evaluation reports that focused on Even Start's outcomes, read those reports carefully, and used the data and findings from those reports in the descriptions in Section 5. The reports included 3 state reports with outcome data, 3 local reports based on a non-equivalent comparison group design, and 18 reports based on a one-group pre-post design with reasonably large samples and measurement of both adults and children.

Each of the 24 reports addressed the question of program outcomes, and we had to summarize data reported in differing levels of detail from studies that used differing (if overlapping) measures, varied data collection strategies and schedules, and different analytic strategies. In recent years, this type of challenge has increasingly been addressed through formal meta-analysis of cross-study data on program services, activities and implementation, and outcomes. However, coding studies for a meta-analysis is a time-consuming and expensive process which was beyond the resources of this project.

Instead, we prepared a narrative synthesis of findings based on a series of tables created to summarize the salient features of each study, as well as the outcome findings for each study. This strategy is often used in small-scale reviews. The general approach was to create tables for different research questions in which each of the 24 reports is a row and the cells of the table contain summary statements and/or summary data from the studies about the research question. The discussion in Section 5 of this report was drawn from reviewing the evidence contained in these tables.

[2.0 Even Start Evaluation Requirements]

[4.0 Description of Local and State Evaluations]