A r c h i v e d  I n f o r m a t i o n

Longitudinal Study of Family Literacy Program Outcomes

The question, "Are family literacy programs effective?" has no simple answer, although American society often demands simple answers to complex questions. To approach the real complexity logically, one must discover for whom literacy programs are effective and under what conditions effectiveness is assured. To be effective, itself, this paper will begin by defining conceptual terms used throughout, including

• evaluation,

• research, and

• family literacy.

In addition to discussing the above terms and the ideas that give them meaning, some conclusions shall be drawn concerning target families, program operations, and program effects that have implications for evaluation and research design (as well as for data interpretation).

Evaluation

A concept that is central to this definition is another term, information, which, as used here, refers to data that have the capability to reduce the probability of error in a choice among alternatives. Thus, by definition, information is useful for the intended decision-making function; and, to be sound, evaluation designs must reflect directly any decisions to be made, including who will make them and how they will be made.

Research

It seems reasonable that a definition of research should be useful for directing the planning and conducting of a project. For definition, research includes all processes through which information is provided for reducing the probability of error in reaching conclusions regarding the object of the research itself.

Family Literacy

Several recent publications have addressed the question of how to define family literacy (Brizius & Foster, 1993; Harris & Hodges 1995; Morrow, 1995; Morrow, Tracey, & Maxwell, 1995; Morrow, Paratore, & Tracey, 1994; and Tracey, 1995). All studies have indicated the need for such a definition, and some have attempted to fill that need, with different degrees of success. Most attempts, however, have done little to define the concept in terms that could support planning for inquiry to determine changes in family literacy.

To serve the need for planning evaluations or research projects, the definition here focuses on literate families only. This working definition was arrived at by determining the capabilities needed by family members to ". . .communicate expectations of accomplishment to their children," as only one example. The list of capabilities below were derived directly from the several general family characteristics that are considered central to providing a setting or a context that would have desired intergenerational effects be present.

The literate family capability areas seem to be those needed to assure a setting and family conditions that can suffice to support development and learning by children within the family context. Each of these items, when considered alone, seems to be necessary but not sufficient for supporting family development. Furthermore, evidence of change in only one, or a small number, of these items does not indicate that a change in family literacy has occurred, even though the change that did occur may be important to individual members or, for that matter, the whole family. In short, each of these items identifies an area of capability that can be assessed for evaluation of program effects or for research. The definition here used for a literate family is made up of many parts, including the ability or means to

• acquire needed or desired information from printed verbal, symbolic, and graphic materials;

• acquire needed or desired information from oral communications;

• communicate their intent or ideas to others in printed (written) verbal, oral verbal, graphic, or symbolic forms;

• communicate the value and worth of actions and being of self and family members;

• communicate in oral and written forms with content in the traditional academic area at a level needed for family tasks;

• judge the plausibility of data in print, oral, or graphic forms;

• judge the plausibility of goals for self and family;

• set short-term and long-term goals for self and family;

• communicate goals to family members;

• make plans for accomplishment of personal and family goals;

• implement plans for accomplishment of personal or family goals;

• analyze problematic situations, and select and manage solutions to problems for self or family;

• make valid predictions of the probable effects of their actions or of family conditions on themselves and others;

• make valid judgments of the perceptions others hold of their own behaviors or ways of being;

• make valid judgments of the quality of their own work and conditions;

• set and communicate expectations for self and others;

• support the development of family members, and help others in the family with their learning and development; and

• acquire skills needed to make changes in self or family that may be needed or desired.

In any given family or target group, the level of attainment of capabilities at the time of enrollment or at any later time may range from full in all areas to substantial deficits in all, or most areas. Literacy needs, therefore, are any deficits in the individual or family capabilities listed above.

In a further attempt to produce definitions to support evaluation and research design, family literacy programs are distinguished from any of the variety of single-service programs available for adults or children. That being the case, family literacy programs are those intervention systems that

• address one or more of the capabilities required for a literate family;

• share intergenerational program goals with the family; and

• provide or coordinate services to meet family literacy needs in a way that is sufficient to produce intergenerational effects.

Conclusions Reached and Assumptions Used for Evaluation and Research Design

In programs designed to address long-term and intergenerational changes in lives or capabilities of families, the intervention usually produces direct effects that are attainable during relatively short time periods, and in the same categories as those produced by many programs designed to produce only direct, short-term effects. For example, adult-education programs or work-place literacy programs and family literacy programs may produce changes in the education levels of the adults in a family or lead to some form of certification of skill or achievement, but family literacy programs are designed to provide a more complex set of services for adults and children than are programs designed to provide one primary service. The different programs, then, should be expected to have different direct and long-term effects.

While direct, short-term effects of interventions may be of significant value in themselves, for programs designed to produce long-term or intergenerational effects, those effects are instrumental to family changes that may require a long period of time to be manifested--more time, in fact, than duration of their participation in a program. The multiple services with long-term goals included in such programs may cause them to be less efficient in producing direct, short-term effects than programs with single, direct-effect goals. Thus, when compared on General Educational Development (GED) production, job placements, or other direct adult-education goals, family literacy programs may show less effects than single-service programs. However, it is expected that the combined effects of the family literacy program services will be greater in the long term than in single-service systems.

Because of the traditional short-term funding patterns and political decision time frames for special programs, care must be taken in program design and operation to make legitimate the goals of both long-term or intergenerational intervention and short-term indicators of changes in the family context or family culture, particularly those that are causally related to conditions in the lives of the children as they grow into adulthood. Without this care, the programs with long-term and family effect goals may lose out compared to single-service or single-member intervention programs and services.

Some Points to Consider in Framing Expectations for Long-term Program Effects

The points made in this section describe factors that need to be considered in both setting long-term goals and planning evaluation or research projects. These ideas deal with a variety of factors, some of which have implications for methods, some for variable selection, and some for interpretation of results. Numbered not necessarily according to impact, the points are:

1. Correlational studies (as opposed to causal studies), reported in various professional publications and often quoted glibly in the popular press, have led to many overly simple conclusions about ways to intervene in order to address literacy as well as other conditions of society. While these findings are not incorrect, all are significantly inadequate for framing public policy or for reaching conclusions about causal influences. Yet each finding has been used as a major rationale for direct intervention. Among these findings are:
   • Reading material in the home is related to reading performance.

   • Education levels of parents are related to a child's later academic success.

   • The amount of time parents spend reading to their children is related to the later academic performance of those children.

   • The amount of time that children spend watching TV is related negatively to reading and academic performance.

   • Children of single-parent families perform less well than children from two-parent families.

   All of these correlational studies and their conclusions suffer from what is considered to be a fatal flaw--inappropriate variables were identified by the studies. The variables actually identified are symptomatic of the causal variables, but are not the causal variables, themselves. In all cases, elements of the "family culture" are the causal factors; and those cultural elements are revealed by, or manifested in, the behaviors or conditions that were studied. However, the manifestations, or symptoms, were described as the causes and thus assumed by many program planners and advocates to be the causes.

   There is little reason to believe that introducing any one of these conditions into family settings through intervention will have the same effects as when they are present in the family context as a result of their having been introduced into the family as a normal and natural action by the family.

2. The adults who currently comprise a major portion of the U.S.'s undereducated class represent families with histories of undereduca-tion, underclass identity, poverty, and so on. Those characteristics are part of the system of meaning of the families--part of the family culture. Significant changes in these families through external intervention probably will not occur without significant changes in the system of meaning experienced by members. To be effective, interventions expecting to change that culturally-based system of meaning must be as complex, intensive, and of such duration as is needed by the particular target families to make significant changes.

3. Even under the worst of social conditions and from the most unfavorable family contexts, a large proportion of children turn out OK. Conversely, even under the best of conditions and the most favorable family cultures, many children do not.

4. A large proportion of adults who enroll in family literacy programs had unhappy school experiences. Even if they consider education to be important, they may not think of schools as good places to be. Adult programs that are primarily academic in nature, or which are delivered in settings other than schools, often do little to change the view of K-12 schools and schooling that is based on their prior experiences as a student in those schools.

5. Adult-education programs designed primarily to prepare for GED or other equivalency certification can cause changes with intergenera-tional effects on the lives of families. Whether or not those effects occur depends, of course, upon conditions within the family and upon changes adults make in the family context as a result of their own education.

6. Existing academically oriented adult education programs are not effective for large percentages of the undereducated adults in society. A large proportion (perhaps well over 50 percent) of those persons who enroll do not remain in the various programs, and a large proportion who remain make little or no significant changes in their life states after participation. Data on changes and lasting effects generally are weak.

7. The set of adults who enroll in family literacy programs is not homogeneous. The range of academic functioning at the time of enrollment may be from minimal literacy to near passage of certification exams, or beyond. Families vary, too, according to degree of acceptance of social norms, commitment to change, hopefulness in self and personal conditions, confidence of change, personal capability to learn and change, and supportiveness of their environment. Both short- and long-term family conditions and effects are different among these groups, so there should be different expectations of both short- and long-term effects.

8. Before the age of three or four the cognitive and behavioral patterns of children reflect conditions and values of the home setting. Intervention with the family can cause patterns of behavior to change and can cause expressions of worth and value made by parents and children to change; but even by this early age, the interventions must counter significant patterns of behavior.

9. The social maturity of a large proportion of adults who enroll in family-literacy programs is delayed. In dealing with matters of life, how to solve personal and family problems, and in talking about the future, those adults demonstrate behaviors, views, and values that are similar to those of children during their early-adolescent years. Expectations for types and levels of short- and long-term changes must reflect those levels of adult social maturity.

10. A significantly large proportion of parents who enroll in family literacy programs show little initial understanding of their roles as parents, and especially the role of teacher.

11. Almost all preschool-age children enrolling in family literacy programs enjoy attending school, and enjoy being in school with their parents, but many (especially boys) seem to value school attendance during that age primarily because it represents their being "grown up."

Designing and Conducting Longitudinal Evaluation or Research Projects

It should be clear from the sections above that if longitudinal studies of family literacy program effects and conditions are being considered, some decisions apparently can be based on the information produced. When the question about decisions to be made is raised, however, the responses are either: "We just want to know whether the programs are effective," or "We want to be able to make a case with (whatever the funding agent) so that funding will continue or increase."

Both of these responses are suspect as justification for longitudinal study projects for a variety of reasons that should be apparent from the presentation above. Furthermore, these reasons do not provide the evaluators or researchers with the information needed to design and conduct the project.

"We just want to know" is neither a decision nor a question. If this statement actually means, "To what extent did we achieve some particular objectives?" then the decisions to be made or questions that might be answered with that information need to be identified and made explicit to the sponsors and investigators. As an example, some different decisions that might be based on this information are ones related to program changes, target-audience changes, or changes in the priorities of the organization. These, and other classes of decisions, would require different information from the study. Thus, the decisions to be supported must be made explicit at the time of design to prevent either under- or over-production of information and under- or over-allocation of resources.

"We want to get continued, or expanded funding" as a rationale for longitudinal studies generally ignores common processes and rationales for much of public policy decision-making. We often hear about the objectivity and rationality of public policy decision-making as a justification for project evaluations, but my experiences during over 30 years of planning and evaluation work in almost all states, several federal agencies, and in many communities indicate much the opposite. While there certainly are many important exceptions, the basis for decision and action is much more likely to be advocacy, incidental and anecdotal evidence, case presentation, political agendas, value systems, or general perceptions than objective evidence of program effectiveness or efficiency. Furthermore, it is unusual for policy decisions to be delayed until the longitudinal evidence is available.

What, then, are some reasons for longitudinal studies of family-literacy program effects? That question must be answered before engaging in the processes themselves.

Conditions Required for Justification of Longitudinal Evaluation or Research

Before the question of long-term effects is addressed for making decisions or answering questions, there are some standards of another type that should be considered to determine whether longitudinal studies can be justified. These are all matters that should be determined as a result of evaluation and monitoring during the operational period or in the short term afterwards, including:

1. There must be some defined and documented target audience for the program, and that audience must have some demonstrated need for assistance that can be served by the program as it was designed.

2. There must be some known (documented) program that the subjects actually experienced.

3. The design of the program or project must be consistent with sound theoretical principles for changing the attributes of families that are identified for change. The design should specify, among other things, what changes are expected to occur, the assumptions about what causes those changes to occur, and the rationale for the designed experiences as vehicles for the intended changes. If the design makes little or no sense theoretically as a way to bring about the intended changes, or if the intensity or duration of the experiences are inadequate to cause the changes desired, then there can be little justification for allocation of resources to longitudinal studies of effects.

4. There should be some important question or questions that need to be answered about the program effects for which answers are not adequately available in existing literature or obtainable from short-term studies. If sound theoretical and technical information bases exist in literature for making the decisions in project and program design, there is little need to try to prove the validity of the theoretical or technical frameworks by documenting the outcomes of their application. If, on the other hand, the model designs are based on hypotheses of causes-and-effects, such studies may be justified. If there are sound principles supporting the designs, accountability decisions may only require evidence of implementation as designed. If more evidence is needed, comparison of short-term effects with those predicted may be useful.

5. There should be ample evidence that the designed model was the actual program implemented, or that the program actually implemented was documented to such an extent that it can be tested for theoretical validity and then used to explain effects and their conditions.

6. Any short-term effects that are believed to be necessary conditions for long-term changes must have occurred to the degree essential to expect the long-term effects to follow. Short-term evidence must support claims that direct, short-term effects actually occurred of the form and to the extent required by the model used to explain how long-term effects occur.

7. Some decisions or questions of enough consequence for justification of the resources required should be explicit and known by the sponsor and investigators. The particular questions to be answered, or criterion-information needs should be known to them.

8. Data bases documenting variations in model implementation among sites and years of operation, and documenting the individual participants and their short-term accomplishments must be available in a form that can be used by the investigators.

9. The identities of the participants and some information about how to contact them must be available in a form that can be provided to the investigators.

10. Relationships with participants that support ongoing contact with them, and that support gathering information from and about them, should have been established during the time of their participation.

11. Resources, including money and technical capability, required to obtain information of the forms, quantity, and quality needed, should be available.

Dangers of Longitudinal Studies of Family Literacy Program Effects

If family literacy program models have valid designs and are effective when implemented well, and if evaluation or research projects are important, then it is essential that the information produced by the investigation provide evidence of the validity of the model and of its outcomes.

Data obtained about program effects and conditions leading to those effects are subject to any of a variety of measurement tools, design, and execution problems. There are many well-documented forms of data error that may indicate that programs were not effective or did not function as hypothesized, when, in fact, they were or did function well. Longitudinal evaluation or research projects should not begin unless planners are confident in the design of the project. Among the processes that may, however, produce no-win outcomes are the following:

1. Comparing small groups of family literacy participants in "comparable" groups who participate in other programs, or some who are in no program at all. The differences among the participants within given family literacy programs generally are so great that group statistics--average values, for example--may be inappropriate to use for interpretation or comparison. Direct, statistical comparisons with almost any groups are likely to be suspect because of non-equivalence, whether the results are positive or negative in respect to the question or purpose intended.

2. Using effects variables that are likely to be affected significantly by factors other than ones that are under control of the participants, or using ones that are situationally inappropriate. Counting the jobs obtained by people who were formerly on welfare may not reflect the changes--for better or for worse--of the general economic conditions or job market in an area. Counting enrollment in other education programs, or entry into the job market might be misleading for the many women who have young children at home, and are staying at home to care for them until they enter kindergarten.

3. Using tests or other measuring devices that are subject to errors because of social desirability or any of several other forms of error; especially if the data are used as if the errors did not occur. All forms of measurement or methods of data-collection are subject to various forms of error, even such forms as interviewing or observing behaviors. All measurement and data-collection materials and processes must be as free of these known forms of error as reasonable to expect in the design.

4. Conducting economic-impact analyses and reporting data in aggregate form as if the conditions and effects for all families served are similar. While data in this form tends to be "sexy," easy to quote, and "headline ready," when presented they are almost always gross overstatements of effectiveness.

5. Identifying the wrong variables --symptoms of, and not the actual interest--or identifying an inadequate set of variables to explain changes or lack thereof, to study and document. Any research relating to intergenerational effects which is for purposes other than hypothesis-development or prediction should address causal variables which help us understand the systems of meaning comprising the culture of homes, and the variable set must be of enough size to comprise a sufficient set to explain variance in effects or conditions. Continuing to address correlations between life states and symptoms or manifestations of social contexts is distracting to the fields of practice.

6. Planning the project for time periods that are either too short or too long to be optimum for providing the information needed.

7. Setting expectations of program effects that are too high (or too low), and then expecting the longitudinal study to document these effects as if they actually should occur. (Nickse, 1993).

8. Failing to specify clear purposes for the evaluation or research project.

9. Using inquiry processes that either have significant intervention effects or that create significant barriers to data gathering and follow-up. For example, using participants in data collection is an unnatural activity for the family, and would be a form of continuing intervention; and asking participants to keep records of their activities may be both a significant intervention and barrier to their continued participation in the longitudinal study.

10. Using amount of time enrolled in the program as a factor related to program outcome. This variable can have little meaning as a factor indicating effectiveness of programs without accounting for all of a variety of variables on which the participants are different, including, among others, entry educational level and capability, goal aspirations, or supportiveness of environment.

Some Purposes for Longitudinal Study

It may seem that the sections above are suggesting that there is little reason to do any longitudinal evaluations or research. Perhaps in a way they are. However, it is wise to remember that evaluation must provide information for decision-making, and research is to provide information that makes answering questions less prone to error. If there are important decisions or questions that probably will be based on the forms of data that can be provided only by longitudinal study, then such studies are essential. If, on the other hand, decisions or conclusions are not going to be based on such information, then resources should be saved--including organizational energy.

For family literacy programs and their related target audiences, there are many unanswered questions about needs, techniques of intervention, systems of services, differences among audiences that justify differences in programs and techniques, and interactions of interventions with other societal factors or systems. These questions may be important to staffs and other decision makers within particular program sites, or within the field in general.

If longitudinal studies are primarily to serve research purposes, they should be justified and designed toward that end. Quite often, research information must take a different form from evaluation information, especially if the decisions to be made are concerned primarily with policy or advocacy. Whatever the form, both the level of informative detail and the form such detail takes will differ significantly.

Some Approaches for Longitudinal Study

It should be clear from the above that there can be no best plan or design for longitudinal study, for the design will always depend upon the purpose to be served. Some approaches to evaluation or research that may fit the information needs for the different conditions are illustrated below. These might be combined in many ways, or modified to fit situations that are different.

1. Start the longitudinal study at the time of planning the family literacy project and the short-term evaluation. Plan to produce a base-line data set that can justify attributing later conditions to program effects or conditions and can demonstrate, also, that changes actually have occurred.

2. Use case-documentation data systems that are designed to allow all appropriate aggregation of case data, including a full range of objective and subjective data. Begin case documentation at the time the subjects are identified, and continue throughout program participation and longitudinal study.

3. For projects with one, or only a few sites, base evaluation studies on case methods, using some system to classify cases into categories that are reasonably related to goals, needs, and family conditions. Interpret short- and long-term effects by category.

4. Document the variations in model implementation each year (or for each client cohort), and plan to consider any significant variations in the interpretations of data.

5. For projects with multiple sites, use a system of best- and worst-case analysis in which participating families are selected from among the sites. These would be the cases we might all agree are "the ones that really worked or didn't work". Use investigative methods traditional in epidemiological studies to determine the reasons for success or failure. These cases may be identified at any time in their history with the project at which success or failure can be established, and should be followed as long as useful. The cases can be selected from among the types identified in #3 above.

6. Use staff from a local site who were especially successful in establishing relationships with participants to maintain the client-locator component of the longitudinal study, and use periodic group or individual interviews with site staff to obtain any information they have about former participants.

7. To determine how children perform in schools, use a system in which current teachers are interviewed in person or by telephone. In this process, ask the teacher to provide information about two students-- the one of interest, and one selected randomly by picking a number in the range between one (1) and the highest number of students likely to be on a class roll. This second child provides a randomly selected comparison group, giving a normal referent. Furthermore, it provides a way to control for social desirability distortions in the responses made by the teachers.

8. Use deliberate sampling methods to select small sets of families who will be the object of intensive case study. These samples may be by family type (#3 above), success stories, or any other useful types, such as representative of a population for which there are some particular interests.

9. At each contact point, obtain as much situational data for the families as feasible to provide foundations for interpretations of later conditions.

Summary

Longitudinal studies of families who participate in family literacy programs should be purposive in nature, and should reflect the variety of technical and theoretical information available about families, political and professional decision-making processes and criteria, cultural and social phenomena and their potential for change, and information-systems design and operation. The varieties of needs, goals, and family conditions among the participants should be used deliberately as factors in data analyses and interpretations; and this variance should not be perceived merely as a problem to be solved by ignoring it or trying to work around it because of its problematic nature.

Studies should be grounded in a sound theoretical design of the program model, which is the object of question, documentation of model implementation, documentation of the short-term effects of model implementation, and documentation of the cases to be studied. Longitudinal studies, after all, require too many resources and too much organizational energy to be conducted in any way but to obtain effects of the highest possible quality.

There seems to be little question at this point in the history of the field of family literacy, however, that several compelling questions must be answered through longitudinal study to provide needed focus to the field, to support policy action, to make program-design decisions, and to resist the attacks of critics or others who would use the field as a strategy to gain support for their own interests. Since family literacy program goals are long-term and intergenerational, there are several significant needs for information about how, and to what extent, those goals are, or can be achieved. It will become more difficult over time to advocate for family literacy programs without evidence that important and long-term goals are being achieved by given target populations through family literacy programs that are not being achieved through other intervention systems.

References

Brizius, J.A. & Foster, S.A. (1993). Generation to generation: Realizing the promise of family literacy. Ypsilanti, MI: High/Scope Press.

Harris, T.L. & Hodges, R.E. (Eds.) (1995). The literacy dictionary: A vocabulary of reading and writing. Newark, DE: International Reading Association.

Morrow, L.M., Paratore, J.R. & Tracey, D.H. (1994). Family literacy: New perspectives, new opportunities. Newark, DE: International Reading Association.

Morrow, L.M. (Ed.) (1995). Family literacy: Connections in schools and communities. Newark, DE: International Reading Association.

Morrow, L.M., Tracey, D.H. & Maxwell, C.M. (Eds.) (1995). A survey of family literacy in the United States. Newark, DE: International Reading Association.

Nickse, R. (1993). A typology of family and intergenerational literacy programs: Implications for evaluation. ERIC Document Reproduction Service No. ED 362 766.

Stufflebeam, D.L. et al. (PDK National Study Committee on Evaluation) (1971). Educational evaluation and decision making. Itasca, IL: F.E. Peacock Publishers, Inc.

Tracey, D.H. (in press). Family literacy: Overview and synthesis of an ERIC search. Yearbook of The National Reading Conference. NRC.

-###-

[Commissioned Papers: Creating...]

[Commissioned Papers: Parent and Child...]