A r c h i v e d  I n f o r m a t i o n

National Evaluation of The Even Start Family Literacy Program, 1998

Chapter 7

What Were the Adult Education Outcomes?

Projects in the Sample Study have been able to choose either the CASAS or the TABE as a measure of adult math and reading skills. Projects could choose either one or both measures to assess their adult participants' progress. The choices reflected a match between the test and the curricular orientation of their adult education programs (e.g., the CASAS may be a better fit for projects that emphasize functional literacy while the TABE represents a better fit for projects that offer a more academic or GED preparation emphasis). Some projects chose to administer different tests to different students based on their skill levels (CASAS for lower level students, TABE for more advanced students). Giving projects the option to administer either the TABE or the CASAS represents a change from the earlier Even Start evaluation, as does the addition of the math tests (for both CASAS and TABE). Only the CASAS reading test was used in the first evaluation.

Comprehensive Adult Student Assessment System (CASAS)

The Comprehensive Adult Student Assessment System (CASAS) is an adult-oriented functional assessment system that measures a broad range of adult literacy skills and their application in real-life domains, including consumer economics, government and law, occupational knowledge, community resources, and health (Rickard et al., 1990). For this evaluation, projects administered the CASAS Life Skills tests in both reading and math.

The CASAS has the flexibility to measure participants involved in diverse adult education programs, spanning the range from non-readers to adults at the high school level. An untimed paper-and-pencil test, each CASAS Life Skills test may take as long as sixty minutes to complete. The CASAS has been used with adult education learners in twenty-seven states. The test is used in adult education and job training programs with both native and non-native English speakers. The CASAS has been used in the National Evaluation of Adult Education Programs (Development Associates, 1992), in the evaluation of California's GAIN program (CASAS, 1990), and in the evaluation of California's 321 adult education programs (CASAS, 1991).

CASAS scores range from 150 to 260. Scale scores link the levels into a continuous scale of achievement. The test developers suggest the following interpretation of CASAS scale scores:

Beginning literacy (below 200): Adults scoring below a scale score of 200 have difficulty with the basic literacy skills needed to function in an employment setting and in the community. While these adults can handle routine entry-level jobs, they may have trouble following simple directions and safety procedures.

Basic literacy (200 through 214): Adults scoring between scale scores 200 and 214 can function in entry-level jobs that require only minimal literacy skills. They can complete simple application forms.

Intermediate literacy (215 through 224): Adults scoring between scale scores 215 and 224 are able to perform basic literacy tasks in an employment setting. They are generally able to function in jobs or job training that involves following written instructions and diagrams, although they usually have trouble following complex sets of directions.

High school literacy (225 and above): Adults scoring above a scale score of 224 can usually perform work that involves written directions in familiar and some unfamiliar situations. They generally can function at a high school entry level in basic reading. If they do not have a high school diploma, they can benefit from instruction in GED classes and have a high probability of passing the GED test in a short time.

Sticht (1990) found these interpretations to be reasonable and reported general correspondence between CASAS scale scores above 225 and the 9th- to 12th-grade reading levels on the Tests of Adult Basic Education (TABE) and the Adult Basic Learning Examination (ABLE).

There are only minimal data on the psychometric characteristics of the CASAS. A correlation of .70 between the CASAS reading test and the ABLE was reported in unpublished data. In the earlier Even Start evaluation, using data from the NEIS, an estimate of test-retest reliability was calculated using the correlation between pretest and posttest scores for adults who were posttested less than ninety days after the pretest. The correlation was .86, suggesting that the CASAS is a reliable measure. The true test-retest reliability may be even higher since this estimate is based on data using alternate forms of CASAS tests.

In the Sample Study, Even Start staff administered the CASAS Life Skills tests in reading and math. Each test has four levels, A through D, with twenty-four to forty items per level and alternate forms of each level. Staff administered a short "appraisal" test to assist in identifying the appropriate level of the CASAS. There is no Spanish version of the CASAS, and project staff were instructed to administer the sample items on the appraisal test to determine whether adults had enough ability in reading English to take the test. If an adult was not given the test due to limited English proficiency, this was noted on the ESIS form.

Performance on the CASAS

Because we have relatively little additional information on adults from the 1996-97 program year, the findings reported below (on pretests as well as pretest-to-posttest gains) are quite similar to what was reported in the 1996 Interim Report. The average pretest scale score on the CASAS reading test was 226, with a standard deviation of 19.6 (see Appendix D, Exhibit D.5). This score corresponds to what CASAS terms an "Advanced Basic Skills" level, which means that the test taker is capable of managing most routine literacy tasks, and test takers at the high end of this range are ready to begin GED preparation. Pretest means increased as the parents' education level went up, from 186 for parents with less than a 5th-grade education to 229 for parents with some high school. The small group with a high school diploma or GED certificate scored slightly higher than those with some high school education. Not surprisingly, the average scores of native English speaking parents were higher, on average, by nearly two standard deviations than those whose primary language is not English (234 versus 208).

The average pretest score on the CASAS math test was lower than for the reading—219, with a standard deviation of 14.4 (see Appendix D, Exhibit D.6). This score corresponds to an intermediate level of functional skills, indicating that the average Even Start adult's math skills are below the high school level at entry into the program. The pattern observed in reading scores occurs for math scores as well, with higher scores associated with higher education levels. We observed an average score of 189 for adults with less than a 5th-grade education, compared to 218 for those with some high school education. Those with a high school diploma or GED certificate scored about a half standard deviation higher, on average, than those with a 9th- to 12th-grade education (226 versus 218), suggesting that additional education may play a more substantial role in math than in reading skills. This difference is not, however, statistically significant. The difference in math scores between those whose primary language is English (221) and those who are non-native speakers (212) is less striking than the difference in reading scores.

Adults who took the CASAS reading test at both pretest and posttest gained an average of 4.2 scale score points (Exhibit 7.8), equivalent to a gain of .22 standard deviation units. Additionally, adults in the Sample Study gained an average of 5.9 points on the math test from the pretest to the posttest, equivalent to a gain of .44 standard deviation units, approximately six months later. The math gains are nearly half a standard deviation in size, larger than the gain seen for reading. As noted above, initial scores were lower for the math test, allowing more room for change.

When the scaled scores in reading are translated into literacy levels (Exhibit 7.9), we see that the majority of adults (67 percent) were at the high school level at both the pretest and posttest. However, there was some modest movement at the lower levels. For example, 8.8 percent of adults moved from an intermediate level of literacy at the pretest to the high school level at the posttest, and about 5 percent moved from the basic literacy level to the intermediate level.

The levels of math literacy at pretest and posttest are shown in Exhibit 7.10. Approximately 43 percent of the adults scored at the high school level at both the pretest and posttest. Approximately 19 percent of adults moved from an intermediate level of proficiency at the pretest to the high school level, and about 5 percent moved from a basic to an intermediate level.

We explored associations and relationships between the CASAS reading and math tests and key predictors, using the same approach as for the HSQ, that is, testing simple correlations first and subsequently building simple and multiple regression models. We examined relationships with such predictors as pretest scores, the amount of adult education hours, family need index, and several project staff characteristics. As was the case for the HSQ, we found that only pretest score remained a significant predictor of gain score for the CASAS reading test; pretest scores account for 20 percent of the variation in gain scores on the reading measure. For the math test, however, the only significant predictor was the proportion of adult education instructors with at least a Bachelor's degree, which explained 14 percent of the variation in gain scores. Interestingly, the pretest score was not a significant predictor for gains on the CASAS math test (see Exhibit 7.7).

The magnitude of the adult literacy gains in the Sample Study is comparable to the magnitude of gains evident in other adult education programs. The gain of 4.2 points on the CASAS reading test is larger than the gain of 3.6 points observed at the first follow-up on the earlier Even Start evaluation. It is nearly four times larger than the 1.2 point gain observed among the control group in the In-Depth Study from the earlier evaluation. When translated into standard deviation units, .24 in reading for the Sample Study is comparable to the .26 observed with the NEIS data from the first evaluation. However, the analytic sample for the first evaluation was restricted to adults with at least seventy hours of instruction, which was not the case for the Sample Study due to the small sample sizes. The standardized gain for the Sample Study also is statistically equivalent to the gains reported in other adult education evaluations using the CASAS. For example, in an evaluation of federally funded adult education programs in California, researchers found average gains of 3.0 scaled score points and a standardized gain of .20 (CASAS, 1992) when adults were tested after eighty to 100 hours of instruction. The fact that we have observed gains for adults in the Sample Study (where there has not been a minimum amount of instruction, as is the case in other studies), and that the gains are comparable to those observed in studies of populations with higher education levels and less stark poverty, suggest that participation in Even Start is having a positive effect on adults' literacy development.

Tests of Adults Basic Education

The Tests of Adult Basic Education (TABE) are norm-referenced assessments designed to measure achievement in reading, mathematics, language, and spelling. The test items are written to reflect language and content appropriate for adults and to measure the understanding and applications of conventions and principles commonly taught in adult basic education curricula (CTB/McGraw-Hill, 1987). The test has been normed on a sample of adults representing participants in adult basic education programs, postsecondary vocational-technical schools, juvenile correctional facilities, and adult correctional institutions. There are four overlapping levels of the test:

• E (Easy) corresponding to grade levels 2.6 through 4.9;

• M (Medium) corresponding to grade levels 4.6 through 6.9;

• D (Difficult) corresponding to grade levels 6.6 through 8.9; and

• A (Advanced) corresponding to grade levels 8.6 through 12.9.

In the Sample Study, projects administered only the reading and mathematics tests of the TABE. The reading test assesses vocabulary (e.g., synonyms, antonyms, words in context, meaning of prefixes and suffixes) and comprehension (e.g., extracting details from text, analyzing characters, identifying main ideas, and interpreting events). The mathematics test measures computation (e.g., addition, subtraction, multiplication, division, fractions, and percents) and concepts and applications (e.g., numeration, problem solving, measurement, and geometry).

There are two parallel forms for each level of the test. There also is a complete battery as well as a shorter survey version of the tests. The complete battery provides scores in each subtest (e.g., vocabulary and comprehension) as well as a total score; the survey form provides only total scores for each test (e.g., reading and mathematics). Both tests are scored on the same scale, with scores ranging from approximately 450 to 865.

The TABE is a timed test. For the complete battery, the reading tests take approximately an hour, and the math tests are allotted about ninety minutes. The reading and math survey forms have about half of the items of the full battery tests and take about thirty minutes each. Prior to taking either the full battery or the survey form, students are given a locator test to determine the appropriate level of the tests to be administered.

In the Sample Study, Even Start staff or staff at collaborating agencies administered the TABE reading and math tests. They had the choice of the full battery or the survey form. The TABE was administered in English. (Although there is a Spanish language version of the TABE, it was not used in this study). Project staff were instructed to administer the sample items on the locator test to determine whether adults had enough ability in reading English to take the test. If an adult was not given the test due to limited English proficiency, this was noted on the ESIS form.

Performance on the TABE

The results reported for the TABE are also quite similar to what was reported in the 1996 Interim Report, again, because relatively little additional outcome data were submitted for adults for the 1996-97 program year. The average pretest scale score on the TABE reading test was 716.5, with a standard deviation of 81.3 (see Appendix D, Exhibit D.7). This score corresponds to about the end of 5th-grade reading level.¹⁰⁵ As is the case with other measures, pretest means were higher for higher education levels, from 573.5 for parents with less than a 5th-grade education to 718 for parents with some high school. Interestingly, unlike what we observed in the CASAS, the small group with a high school diploma or GED certificate scored slightly lower, on average, than those with some high school education. Parents whose primary language is English had average scores about one-third of a standard deviation higher than those whose primary language is not English (718 versus 693).

The average pretest score on the TABE math test was almost equivalent to the reading—717, with a standard deviation of 86.0 (Appendix D, Exhibit D.8). Math scores, like reading scores, increased with higher education levels, from an average score of 584 for adults with less than a 5th-grade education to 720.5 for parents with some high school education. Again, those with a high school diploma or GED certificate scored slightly below those with a 9th- to 12th-grade education (702 for the diploma/GED group versus 720.5). The small group of adults whose primary language is not English scored slightly lower, on average, than adults whose primary language is English (711.6 versus 717.5).

The sample of adults who took the TABE reading test at both pretest and posttest gained an average of twenty-three scale-score points (Exhibit 7.11). This is a statistically significant difference over the approximately six months between pretest and posttest. These gains are comparable to those on the CASAS, corresponding to a standardized gain of .27 standard deviation units.

As Exhibit 7.11 shows, adults in the Sample Study gained an average of 22.5 points on the TABE math test from the pretest to the posttest approximately six months later, corresponding to a standardized gain of .25 standard deviation units. This gain is more modest than the .44 standard deviation units observed for the adults tested on the CASAS math test.

The national norms on the TABE can be used to provide information about how the reading and math scores of Even Start adults compare to those of other students in adult education programs. In particular, TABE scaled scores can be translated into grade equivalent scores, which have been obtained by calibrating the TABE scores to the norming sample for California Achievement Test (CAT) used with children in elementary and secondary grades. Thus, a grade equivalent score on the TABE of 6.8 means that the test taker's performance is equivalent to the typical CAT performance of students who have completed eight months of the 6th grade.

To look at the grade equivalent scores of the Even Start adults, we split the sample at pretest and posttest into quartiles and computed the grade equivalent scores of students at the twenty-fifth, fiftieth, and seventy-fifth percentile. As Exhibit 7.12 shows, the scores of students at the fiftieth percentile (i.e., the median) on the reading test at pretest correspond to a grade equivalent of 6.8; by posttest, students at the median had a 8.1 grade equivalent score.

Similar growth was evidenced for the math test. In general, for the reading and math tests, students at the median gained between one to one-and-a-half grade levels from pretest to posttest, while students at the seventy-fifth percentile gained two or more grade levels over the same six months. Evidence from other studies suggests that adults move, on average, from one grade level equivalent to the next only after approximately 80 to 100 hours of instruction, so the fact that Even Start adults demonstrated such progress suggests that Even Start is having a decided effect.

Exhibit 7.12: Grade Equivalent Scores Corresponding to TABE
Reading and Math Quartiles at Pretest and Posttest
(1994-95, 1995-96, and 1996-97)¹⁰⁶

We explored associations and relationships between the TABE Reading and Math Tests and key predictors, using the same approach as outlined above for other measures. First we tested simple correlations (see Appendix D, Exhibit D.9) and, where appropriate, we subsequently built simple and multiple regression models. We examined the relationships between such predictors as pretest scores, the amount of adult education hours, family need index, and several project staff characteristics.

A multiple regression that includes both pretest score and one project-level characteristic, the number of adult education instructors, explains 21 percent of the variation in gain scores on the TABE Reading (Exhibit 7.7). No other predictors were significantly related to gain scores, either in simple or multiple regression models.

Posttest scores on the TABE Math test were significantly associated with pretest scores, the number of hours of adult education between the pretest and posttest, and the proportion of adult education instructors with at least a BA degree. These three predictors together account for 18 percent of the variation in posttest scores for adults. This represents the only measure for which the amount of time spent in instruction has a significant effect on the outcome. For the other adult education outcomes, only pretest scores and/or one measure of staff quality (either the proportion of staff with at least a BA or number of adult education instructors) are significant predictors (Exhibit 7.7).

Variation in Participation for Adult Sample Study Participants

As described both in Chapter 2 and earlier in this chapter, the families who remain in Even Start long enough to have completed a posttest (or those with multiple waves) are systematically different from those with only one wave. We examined participation rates in the following two ways: one, by comparing the number of hours of participation for one-wave adults to multi-wave adults, and two, by examining differences in participation for adults by primary language. In both cases, there are pronounced differences. As shown in Exhibit 7.13, the average number of hours of instruction per year is consistently higher for adults with multiple waves of tests than the average for those adults with only one wave. We might expect that participants who remain in the program for longer periods of time have more exposure to available instruction, and we also might hypothesize that increased participation over time would be associated with increases in test scores. This differential is consistent with the findings from the analyses of children?s test scores reported earlier, where wave status serves as a proxy for time in the program. Were there sufficient multi-wave data on the adults, we might expect to see patterns of growth similar to those evident for Even Start children.

The difference in total number of hours of participation by primary language status is also striking, as displayed in Exhibit 7.14. Participants whose native language is English participate for nearly twice as much time, on average, as those whose native language is Spanish. This represents an interesting finding about participation, because the families with two or more waves of data are more likely, on average, to speak languages other than English than those families with only one wave (as described in Chapter 2 and earlier in this chapter), yet the average cumulative exposure to Even Start is lower for non-English-speaking adults. Perhaps for those adults most interested in learning English, Even Start represents only one opportunity to practice English literacy skills, because there are other avenues for learning English. For those non-ESL participants interested in developing reading and math skills, however, Even Start may provide the only opportunities for formal learning of math and reading instruction. The lack of outcome data for non-native English speakers means that we cannot explore comparisons in outcomes. These differentials are certainly worth exploring in other future evaluations of Even Start.

Footnotes:

¹⁰⁵ Although we cannot explain why the adults who took the TABE have lower reading levels, on average, than those who took the CASAS, it is clear that there are some systematic differences between the two groups of adults. On average, those who were assessed with the CASAS, for example, have completed more years of schooling, have lower scores on the need index, are more likely to be African-American, and are more likely to speak Spanish as the primary language at home than those who were assessed with the TABE. Each of these comparisons is statistically significant at the .001 level, using t-tests or chi-square tests of association.

¹⁰⁶ While data from studies of other individual adult education projects do not present reliable data on pretest and posttest assessments, we do know that states are beginning to implement performance standards for participants in adult education programs. In Connecticut, for example, the state recently articulated a standard of 75 hours of instruction corresponding to a one grade-level increase.

-###-

[ What Were the Parenting Outcomes? ]

[ What Were the Outcomes on Other Progress Indicators? ]