The purpose of this study is to review and synthesize research and identify data sources on the effectiveness of discrete educational software for student achievement. The report developed for this task will provide an overview of existing research, identify trends in levels of use and effectiveness, identify gaps where additional information is needed to inform policy and practice, and note models of emerging best practice.

"Discrete educational software" is software designed and used for technology-enhanced education in core instructional areas such as reading or mathematics. Sales of educational software are growing rapidly. From 1996-1998 annual sales of software and online materials specifically designed for instruction increased by 21%, from $473 million to $571 million. Examples of discrete software include software programs such as LightSpan, Jostens, CCC and Reader Rabbit. While relatively "traditional" in design and focus, such software makes up a substantial portion of the educational software market. Becker and Anderson found that nearly 30% of schools report having one or more integrated learning systems -and nearly 50% of the schools with a large number of low income students and high percentages of minority student enrollment have one or more.

Many of these software programs are built on the foundation of work on computer-assisted instruction, integrated learning systems, and related approaches conducted over the past several decades. Meta-analyses of studies of discrete educational software suggest non-trivial impacts on achievement. Various larger scale studies and analyses of large-scale data sets suggest that it may be feasible to attain such gains on a large-scale under at least some circumstances.

However, much of the prior research may be "outdated", reporting results for technologies which are now several "generations" old and that do not adequately reflect the power of current technologies, better software design and increased expertise in how to use it. Most of the existing work has very small samples, and often focuses on a single educational system. Only a handful of studies provide longitudinal data for a school year or longer. Many do not include data on how the technology is actually used for instruction or on teacher qualifications. Many of these studies are case studies and are ad hoc in how the cases are selected rather than providing a theoretical basis for the case study selection.

A review and synthesis of this research and evaluation in the context of broader work on student achievement and broader school improvement is needed to guide policy and practice, as well as to provide a guide to areas where additional research and/or evaluation is needed.

"How effective are commonly used education technology programs for instruction in core curricular areas, such as reading, mathematics, science or social studies?"

Adequately addressing the study question includes addressing at least the following issues:

1. How much access is there to discrete educational software? How does this vary by school poverty level, grade level, and other factors?

2. How is discrete educational software being used? To what extent are different approaches to using technology integrated into the academic instruction and assessment experienced by students? How do technology experiences vary for subgroups of students (e.g., defined on the basis of age and grade level, urbanicity, income level, prior academic achievement, ethnicity, or gender)?

3. To what extent is use of discrete educational software associated with changes in student outcomes? How are levels of access and use related to these changes? How does this vary across grade levels? Do the changes generalize to broader student achievement? Are they sustained over time?

4. What is best practice? To what extent are there emergent models of promising practice?

In addressing these questions, the study addresses issues such as: To what extent is there a rigorous empirical knowledge base in this area? What are the important gaps in information on each of these areas? And, What policy relevant information is needed in the short and longer term to inform policy in these areas?

Conducted by SRI International, the Review of the Evidence on the Effectiveness of Discrete Educational Technology study reviews published research on the effectiveness of discrete educational technology as reported in research published between 1995 and 1999.

The study includes a review of the extant data and literature on the use and effects of discrete educational software. It focuses primarily on software use in core curricular areas, with an emphasis on work reported during the past five years

Findings from the identified research will be entered into the Research Articles Profiling System (RAPS) electronic data base along with information from other studies of educational technology. RAPS will be used to inform the literature review, as well as providing a cumulative source of information on discrete educational technology and other aspects of educational technology.

The study products will include a research guide that provides practical information for educators and others to inform decisions in selecting discrete software for instructional use. This section will include information on how to assess the claims of effectiveness and evidence presented by software vendors and others. This guide will include a check list section that includes items such as samples, timing of study, and so on. The guide will illustrate the use of its checklist with a step by step analysis of an illustrative study. The guide will include a glossary of common statistical terms encountered in such studies.

Two issue summaries which focus on specific issues in promising or best use, appropriate for use by parent groups, teachers and others to inform effective parent involvement in improving student learning will also be developed.

TIMELINE

The evaluation is being conducted from September 2000 through September 2002.