B1

What is scientifically based reading research?

The statute defines scientifically based reading research as the application of rigorous, systematic, and objective procedures to obtain valid knowledge relevant to reading development, reading instruction, and reading difficulties (Section 2252(5)). To meet the statutory definition, the research must:

• employ systematic, empirical methods that draw on observation or experiment;

• involve rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn;

• rely on measurements or observational methods that provide valid data across evaluators and observers and across multiple measurements and observations; and

• have been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review.

B2

What are characteristics of scientifically based reading research?

When reviewing research findings to determine whether the research on which the findings were based met the four criteria specified in the REA (listed in bold below), readers may want to ask themselves questions about how well the research met each of the criteria. Examples of the types of questions that could be asked about each criteria include:

• Use of rigorous, systematic, and empirical methods. Does the work have a solid theoretical or research foundation? Was it carefully designed to avoid biased findings and unwarranted claims of effectiveness? Does the research clearly delineate how the research was conducted, by whom it was conducted, and on whom it was conducted? Does it explain what procedures were followed to avoid spurious findings?

• Adequacy of the data analyses to test the stated hypotheses and justify the general conclusions drawn. Was the research designed to minimize alternative explanations for observed effects? Are the observed effects consistent with the overall conclusions and claims of effectiveness? Does the research present convincing documentation that the observed results were the result of the intervention? Does the research make clear what populations were studied (i.e., does it describe the participants' ages, as well as their demographic, cognitive, academic, and behavioral characteristics) and does it describe to whom the findings can be generalized? Does the study provide a full description of the outcome measures?

• Reliance on measurements or observational methods that provided valid data across evaluators and observers and across multiple measurements and observations. Are the findings based on a single-investigator single-classroom study, or were similar findings observed by multiple investigators in numerous locations? What procedures were in place to minimize researcher biases? Do observed results "hold up" over time? Are the study interventions described in sufficient detail to allow for replicability? Does the research explain how instructional fidelity was ensured and assessed?

• Acceptance by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review. Has the research been carefully reviewed by unbiased individuals who were not part of the research study? Have the findings been subjected to external scrutiny and verification?

B3

What activities must be grounded in scientifically based research?

The primary purpose of the Reading Excellence Act is to improve the reading skills of students and the instructional practices of current teachers (and, as appropriate, other instructional staff) who teach reading, through the use of findings from scientifically based reading research, including findings relating to phonemic awareness, systematic phonics, fluency, and reading comprehension

Not all activities funded by the REA must be based on scientifically based research, although the statute heavily emphasizes the importance of such research in program selection and implementation. The provisions of the statute that specifically refer to activities based on scientifically based reading research are as follows:

The Secretary must:

• Give priority to applications from SEAs whose states have modified, are modifying, or will provide an assurance that they will within 18 months modify the teacher certification requirements for elementary school teachers. The revisions to the certification requirements must increase the training and the methods of teaching reading required for certification as an elementary school teacher to reflect scientifically based reading research. (Note, however, that this provision is not meant to establish a national system of teacher certification.) (Section 2253(c)(2)(C))

• Evaluate SEA applications using a panel which includes experts who provide professional development to teachers and other staff based on scientifically based reading research. (Section 2253(c)(2)(B))

In their applications for funding, state educational agencies must:

• Describe how the SEA will ensure that professional development activities are based on scientifically based reading research (section 2253(b)(2)(B)(II)) and that all subgrantees will use practices based on scientifically based reading research. (Section 2253(b)(2)(B)(vi))

• Provide an assurance that each LEA receiving a subgrant will provide professional development to classroom teachers and other appropriate instructional staff on the teaching of reading based on scientifically based reading research. (Section 2253(b)(2)(C)(i))

• Provide an assurance that each LEA receiving a subgrant will ensure that teachers, other instructional personnel, and tutors providing supplemental reading support for students entering kindergarten or in kindergarten through grade 3 have been appropriately trained using scientifically based reading research. (Section 2253(b)(2)(C)(iv))

When forming REA partnerships, State educational agencies must:

• Give consideration to including a community-based organization that uses scientifically based reading research. (Section 2253((d)(1)(E))

• Consider including a representative of an institution of higher education operating a program of teacher preparation based on scientifically based reading research in the state (section 2253(d)(2)(A)) and a representative of an eligible private nonprofit or for-profit professional development provider that provides instruction based on scientifically based reading research. (Section 2253(d)(2)(C))

When conducting the required program evaluation, the SEA must:

• Contract with an entity that conducts scientifically based reading research. (Section 2259(a)(2))

Local educational agencies applying for Local Reading Improvement (LRI) subgrants must:

• Describe in their applications how they will work with schools selected to receive LRI funding to select one or more programs of reading instruction that have been developed using scientifically based reading research (section 2255(b)(1)(A)) and that they will use supervised individuals, including tutors, who have been appropriately trained using scientifically based reading research to provide additional support for children preparing to enter kindergarten or in kindergarten through grade 3 who are experiencing difficulty reading (section 2255(b)(2)(D)).

• Include in their application an assurance that they will carry out professional development for classroom teachers and other instructional staff on the teaching of reading based on scientifically based reading research. (Section 2255(b)(2)(A))

Local educational agencies receiving Local Reading Improvement (LRI) subgrants must:

• Secure technical and other assistance from a program of reading instruction based on scientifically based reading research. (Section 2255(d)(1)(A)(i))

• Provide professional development activities to teachers and other instructional staff (including training of tutors), using scientifically based reading research. (Section 2255(d)(1)(B))

• Provide additional support to children entering kindergarten and in grades K through 3 who are experiencing difficulty reading using individuals (including tutors) who have been appropriately trained using scientifically based reading research. (Section 2255(d)(1)(H))

• Provide instruction to parents and reading tutors that is based on scientifically based reading research. (Section 2255(d)(1)(F))

LEAs that receive Tutorial Assistance subgrants must:

• Ensure that the tutorial assistance provided uses instructional practices based on scientifically based reading research. (Section 2256(b)(2))

The National Institute for Literacy must:

• Disseminate information on scientifically based reading research. (Sections 2258(a) and (b))

B4

Must research meet all of the characteristics of the REA definition of scientifically based reading research to be considered?

Yes. The statute requires these characteristics.

Readers seeking additional guidance may wish to consult standard references on research methods. One source readers may want to consider is the National Research Council report"Preventing Reading Difficulties in Young Children" (1998). The National Research Council, when conducting a review of reading research, followed basic guidelines for scientific method. The NRC wrote:

B5

What kind of evidence is critical in determining whether instructional programs and professional development activities meet the required standards?

Applicants should review all theoretical and conceptual claims and assumptions that serve as the foundation for each reading strategy, program, or method of instruction. Too often, assumptions that underlie reading instruction are not supported by scientific data.

A clear definition of reliable evidence of effectiveness is critical to the successful selection and implementation of research-based reading improvement strategies.

One approach to assessing the adequacy of professional development activities or instructional programs is to examine the extent to which they vary along four dimensions: the theoretical foundation for the activity or program, how well the activity or program improves student achievement, the conditions required for activity or program implementation, and evidence of replicability. The Department encourages states to consider these four dimensions when they examine evidence of the effectiveness of research-based reading improvement programs and professional development activities that will be implemented by local educational agencies that compete for subgrants. The types of information states may want to consider along these four dimensions include:

• The theoretical or research foundation for the program or activity. Does the program or activity provide a well-developed theory or research findings to explain why a particular program, service, or activity improves students? reading ability?

• Evaluation-based evidence of improvements in students? reading achievement. Does the program or activity provide evidence of educationally significant improvement through reliable measures of student reading before and after implementation of the program or intervention? Does the evidence make clear the magnitude of the improvement, and show that the results are educationally significant (that is, of sufficient magnitude to make a "real" difference in student performance), not just statistically significant?

• Evidence of the conditions required for effective implementation. Does the program explain what it takes to make the program fully operational, including estimates of the cost, in respect to both time and money, of implementation? Does it explain the full costs of the professional development activities, including teachers? time?

• Evidence of replicability. Has the program or activity been successfully implemented in more than one location? Was information provided on the conditions under which the program or activity was replicated: for example, descriptions of the students? ages, educational background and achievement level; classroom and teacher characteristics; or parental and community involvement?

Ideally, evidence would be available across all four of these dimensions for reading improvement programs and professional development activities under consideration. In practice, the quality of the evidence available for each of the four dimensions is likely to vary not only from program to program but also within a particular program. A program might have a very strong theory for why it should work and evidence that it improves student outcomes for some children, but might have only weak evidence of effective replicability. In considering alternatives, states might want to evaluate evidence along a continuum from most rigorous to marginal.

The table in Appendix A is designed to assist states in evaluating the effectiveness of reading programs and teacher development activities proposed by local educational agencies in their applications, or of the criteria for program selection proposed by the local educational agencies. The table poses illustrative questions that states might want to ask when evaluating research-based alternatives. The most effective programs would be those that can provide the most rigorous evidence for each of the four dimensions. Programs and activities that do not provide rigorous evidence along each of the four would be weaker, with those providing limited information along two or more dimensions being unlikely to meet the REA criteria for scientific rigor. Following Appendix A's table are examples in which the factors are applied to hypothetical reading programs.

B6

Is a state educational agency responsible for ensuring that it funds research-based programs of reading instruction?

Yes. In its application to the Department for Reading Excellence Act funding, each SEA must describe how it will ensure that the subgrantees will use practices based on scientifically based reading research. In determining which local educational agencies receive subgrants, it is the SEA?s responsibility to ensure that all REA-funded programs, services, and activities proposed and implemented by districts meet the requirements of the REA.

States may want to consider using the guidance and chart on research-based programs (see Appendix A) to establish a subgrant process and selection criteria that address this fundamental program requirement. However, use of this guidance is optional.

B7

Must funded local educational agencies select only programs of reading instruction that have been fully evaluated and had the results published in a peer-reviewed journal?

The law requires that local educational agencies work with schools receiving assistance to select one or more programs of reading instruction developed using scientifically based reading research. Local educational agencies may wish to consider whether well-established, nationally-known programs with strong evaluation evidence meet their needs. We expect that many will want to use such programs. However, when selecting the programs, state and local personnel should review the programs carefully to ensure that they meet the criteria for scientifically based reading research in the REA.

A locally developed program that is based on high quality, peer-reviewed research may be acceptable even if the program itself has not been fully evaluated, peer-reviewed, and published in a journal. We encourage states and districts that have identified high quality local programs to fully evaluate the effectiveness of the programs and publish or otherwise disseminate the findings. Again, personnel will need to ensure that the programs meet the REA criteria for reading research.

B8

Where can I get help about what constitutes scientifically based reading research?

In addition to carrying out the broader dissemination activities as described in the Reading Excellence Act (see section 2258), the National Institute for Literacy (NIFL) will provide help to any SEA that requests assistance in determining what constitutes scientifically based reading research. This information could be useful to the SEA in developing its own application and in designing subgrant application forms.

The National Institute for Literacy is an independent federal organization that is leading the national effort toward a fully literate America. By fostering collaboration and innovation, the Institute builds and strengthens state, regional, and national literacy infrastructures, with the goal of ensuring that all Americans with literacy needs receive the high-quality education and basic skills services necessary to achieving success in the workplace, family, and community. Information on NIFL is available on-line at http://www.nifl.gov/ or at (202) 632-1500.

B9

Where can I learn more about reading research that meets the criteria for inclusion in programs and activities funded by the Reading Excellence Act?

Note: Many organizations, including the Department of Education, provide information on a wide variety of programs and practices, ranging from those that have been carefully evaluated to those that are thought to be promising. States should evaluate each program under consideration to determine whether it meets the requirements of the REA. We have provided the following information as a service to readers and not as an endorsement.

For more information, readers may wish to consider the following sources, among others.

• Preventing Reading Difficulties in Young Children. Catherine E. Snow, M. Susan Burns, and Peg Griffin, Ed. National Research Council, National Academy Press, Washington, DC, 1998. http://www.nap.edu/

• America Reads Challenge, the U.S. Department of Education program to improve reading. http://www.ed.gov/inits/americareads/

• The National Reading Panel Progress Report. The National Institute of Child Health and Human Development, National Institutes of Health, U.S. Department of Health and Human Services, February 22, 1999.

• Beginning to Read: Thinking and Learning about Print, Marilyn Adams, MIT Press, 1990.

• Starting Out Right: A Guide to Promoting Children?s Reading Success. M. Susan Burns, Peg Griffin, and Catherine Snow, Ed. National Research Council, National Academy Press, Washington, DC, 1998. http://www.nap.edu/

Eligible applicants may also consult the U.S. Department of Education technical assistance and information providers. The Department?s assistance and information providers include:

• Comprehensive Technical Assistance Centers. The U. S. Department of Education funds 15 Comprehensive Regional Assistance Centers that help states, schools districts, schools, tribes, community-based organizations, and other grant recipients with the administration, integration and implementation of programs funded under the Elementary and Secondary Education Act of 1965. The Centers provide comprehensive training and technical assistance to improve teaching and learning and to meet the needs of children served by ESEA programs. More information about Comprehensive Centers is available at www.wested.org/cc/html/ccnetwork.htm.

• Regional Educational Laboratories. The U. S. Department of Education?s Regional Educational Laboratory Program is a network of 10 Regional Labs working to ensure that those involved in educational improvement at the local, state and regional levels have access to the best available research and knowledge from practice. The program is designed to help educators, policymakers, and communities improve schools and help all students attain their full potential. Information about the Regional Educational Lab program is available at www.ed.gov/prog_info/Labs/

• The Educational Resources Information Center (ERIC). ERIC is a distributed national information system designed to provide users with ready access to an extensive body of education-related literature. Supported by the National Library of Education, ERIC encompasses the world's largest and most frequently searched education data base and a network of knowledgeable and helpful subject experts. ERIC features an extensive Internet presence, including the award–winning AskERIC question-answering service and Virtual Library, and the National Parent Information Network. Additional information about ERIC is available at http://eric.ed.gov/ERICWebPortal/Home.portal.

• The Center for the Improvement of Early Reading Achievement (CIERA). CIERA is a national research center funded by the Department of Education. CIERA?s mission is to improve the reading achievement of America?s children by generating and disseminating theoretical, empirical, and practical solutions to persistent problems in the learning and teaching of beginning reading. CIERA information is at http://www.ciera.org/.

For examples of programs of reading instruction identified by other organizations, readers may want to consult the following sources, among others:

• Learning First Alliance. (1998). Every Child Reading: An Action Plan. http://www.learningfirst.org/.

• The International Reading Association, http://www.ira.org and the National Association for the Education of Young Children, http://www.naeyc.org/ have prepared a joint position statement on reading: Overview of Learning to Read and Write: Developmentally Appropriate Practices for Young Children: A joint position of the International Reading Association (IRA) and the National Association for the Education of Young Children (NAEYC), http://www.naeyc.org/about/about_main.htm