 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Special Education Research and Dissemination of Information | Improve the Current Grant Review Process | Improve Federal Collaboration | Invest in Long-Term Research Priorities | Create a Community of Scholars within OSEP | Improve Development and Dissemination of Research Findings | The Importance of Institutions of Higher Education in the Research Process | The Importance of Research in the Implementation of IDEA | Conclusions | Special Education Research Agenda
Special Education Research and Dissemination of Information
In the federal government, the Office of Special Education Programs (OSEP) within the U.S. Department of Education is responsible for funding coordinated research, personnel preparation, technical assistance, support and dissemination of information to benefit children with disabilities of all ages. Much of this coordination and support for research and other activities is conducted through grants awarded competitively through a peer review system. OSEP manages that system in its Research to Practice Division.
OSEP’s research charge is broad because it spans the range of issues affecting the education and development of infants, toddlers, children and youth with disabilities. In addition, consumers of OSEP’s research efforts include parents and their children with disabilities, teachers, administrators, technical assistance providers, developers and other researchers. These activities play a vital role in improving the achievement of children with disabilities, but the impact has been weakened because of inadequate dissemination efforts. Congress and the Department of Education reaffirm their support of this important research and dissemination program. The Commission recommends several fundamental changes be made to OSEP’s research and development efforts.
Recommendation—Change the Current Grant Review Process to Create Scientific Rigor. Improve the Office of Special Education Programs (OSEP) to make participation in review activities an honor, obligation and a sign of accomplishment among researchers and practitioners. Create a culture of scientific rigor in OSEP emphasizing the quality of special education research activities.
Recommendation—Improve the Coordination of Special Education Research. Integrate and improve the coordination of all research activities within the U.S. Department of Education’s Office of Special Education and Rehabilitative Services. The three offices within OSERS—the Rehabilitation Services Administration, the National Institute on Disability and Rehabilitation Research and the Office of Special Education Programs—must collaborate more effectively with each other and with other federal efforts to improve research related to individuals with disabilities.
Recommendation—Support Long-Term Research Priorities. Focus research investments on a narrower range of priorities to promote the development of more powerful and reliable discoveries that will benefit people with disabilities.
Recommendation—Improve the Impact of Research Findings. Support demonstration and dissemination programs in OSERS that focus on the adoption of scientifically based practices in the preparation of and continuing education for teachers. Focus on proven, effective practices that can be implemented, scaled and sustained nationwide.
Improve the Current Grant Review Process
The Commission finds that OSEP’s current process for reviewing grant proposals and monitoring practices must be improved. The review process at OSEP appears oriented to completion of an administrative task designed to allocate resources as opposed to an opportunity to rigorously evaluate and improve the quality of research. The criteria applied in the review process must match those of the field for rigorous, scientifically based special education research.
Many of the nation’s leading special education researchers provided testimony before the Commission expressing concern over OSEP’s methods of conducting peer review of research grant proposals.77 In addition, several members of the Commission have research backgrounds with years of experience in federal grant application with multiple research agencies and have supported the entire Commission as it developed its findings and recommendations.
Peer review at OSEP is based on ad hoc panels assembled from reviewer lists. The development of a community organized around a set review process, expectations around a set review process, expectations around deadlines for review or the need to participate in review has not occurred. Thus, OSEP experiences difficulties recruiting reviewers. Witnesses indicated that reviewers are often not provided the applications in advance but are expected to review on the spot at the review session. Not only does this prevent careful consideration of the applications, but also written feedback is truncated, inconsistent and often not helpful to the applicant. As a major function of review is to provide feedback to the field and upgrade the quality of research through the review process, OSEP is missing opportunities to enhance the technical quality of its applications.78 This approach also reduces interest in participating in the review process, contributing to the difficulties OSEP experiences in attracting reviewers.
Other problems with the process reflect review procedures for reviewing proposals submitted. OSEP staff persons who are research managers are also responsible for peer review. Not only does this create additional work, but it also creates potential difficulties in separating roles. Review works best when program and review are separated. The review process requires project staff that has appropriate professional backgrounds and experiences in research and its administration, which is not sufficient at OSEP and not apparent in the contracting organizations. The Commission learned that continuation grants are rarely allowed at OSEP, so evaluations of progress rarely figure into the evaluation process. Although OSEP has improved procedures for review, the Commission found little evidence that the results of these reviews figured into future funding decisions for the applicant. Review panels are expected by statute to include researchers, consumers, practitioners and parents of students with disabilities, even though non-researchers may not be able to address technical aspects of the proposal. Many qualified researchers do not participate in the peer review process or avoid OSEP applications because of the perception that the review process is arbitrary and that OSEP does not select the highest-quality grants.
The OSEP review process can be substantially strengthened.79 Setting priorities for research and determining the questions to be addressed in special competitions should be conducted in collaboration with the consumers of special education research—culturally diverse families, individuals with disabilities, service providers, researchers and policy makers. Researchers with methodological and content area expertise that matches the purpose of the competition should be specifically recruited to review research proposals. We propose four broad improvements to create a culture of rigorous scientific practice:
1. The Commission recommends that OSEP develop a peer review system with a two-tiered level of review, which the Commission finds is essential to enhanced research quality at OSEP. A statutory change will be needed to effect this change. The first level should be for technical quality, significance and innovation, completed by members of the research community. The second level should address relevance to OSEP priorities, but it should occur at the level of the assistant secretary for OSERS to ensure the Part D program is coordinated with Part B and C (as designed) and that OSEP research priorities are coordinated with those of the Rehabilitation Services Administration (RSA) and the National Institute on Disability and Rehabilitation Research (NIDRR).
2. A national advisory committee analogous to the National Research Priorities Board at the U.S. Department of Education’s Office of Educational Research and Improvement, the National Science Board at the National Science Foundation (NSF) or the National Advisory Councils at different National Institutes of Health (NIH) institutes should be formed. This committee would include practitioners, researchers, parents and people with disabilities. It would be responsible for helping to establish priorities and agendas and also to review research recommended for funding to ensure its relevance to people with disabilities. Finally, the Commission notes that these concerns about the review process extend to each of the Part D programs under National Activities to Improve Outcomes for Children with Disabilities.
3. To facilitate the first level of review, standing panels with fixed terms for each of the OSEP Part D programs should be established. These panels should operate independently of the OSEP research programs through a separate “institute for review.” This model is like that used by NIH internal review groups that are responsible for research generated outside the investigator-initiated mechanisms but specific to institute priorities.
4. Each panel should be chaired by a senior researcher and administrated by a doctoral level individual with a background in research who is part of the review institute. The administrator will be responsible for processing grant applications, distributing them to reviewers in a timely manner and editing reviews into a consistent format devoid of ad hominem or impertinent comments. Reviewers will be expected to prepare reviews in advance of the meeting that are oriented toward identifying the strongest applications and to provide reviews that are systematic, thorough and document the strengths and weaknesses of the application in an attempt to upgrade the quality of the research. The goal would be to make participation in review an honor, obligation and sign of accomplishment as part of the development of a culture of science around Part D programs, which presently does not exist. The costs of these changes are not significant. Current statute allows OSEP to spend up to five percent in peer review, whereas current expenditures are now about two percent.80
The peer review process itself must be organized in a manner that actively encourages progressive improvement of research proposals through revision and resubmission based on feedback from the previous review. Strong peer review results in better science. More accurate scientific information is needed to improve practice. The review process must promote long-term programs of research that support evidence-based practices. A rational system of review that includes progress reports from previously funded projects and places newly proposed research in the context of the researcher’s record of empirical work is needed. Specific recommendations are as follows:
1. Provide professional, accurate, timely feedback to applicants. The content of the feedback should be substantive and reflect technical adequacy of the proposal in precise terms and delineate the importance of proposed research within the parameters of the competition.
2. Develop a system of grant reviewing that allows for systematic revision and resubmission of proposals, particularly in the Field Initiated Research competition. Reviewers should be provided applications in advance and be expected to prepare reviews that are structured to identifying the strongest applications.
3. Develop standing dates for annual competitions and predictable submission deadlines for special competitions; provide sufficient public notice for applicants to prepare relevant and rigorous applications.
4. Time the reviews and notification of applicants about review outcomes to coincide with the functional start dates for research and training activities. This will allow a more effective and manageable flow of subsequent research implementation and personnel preparation grant awards.
In sum, evidence-based practice depends on sustained funding for important, credible and methodologically rigorous research in special education. A stronger system of peer review can strengthen the methods and standards for research in special education programs and services. Ultimately, a stronger peer review process will increase the reliability and the validity of research outcomes. Credible peer review processes will have a positive effect on special education research as viewed by researchers in other disciplines and by other federal and private funding agencies.81
The Office of Special Education and Rehabilitative Services comprises the Rehabilitation Services Administration, the National Institute on Disability and Rehabilitation Research and the Office of Special Education Programs. Each of these agencies administers research and dissemination activities ranging from rehabilitation engineering technology for people with severe physical disabilities to infant and toddler early intervention programs. Yet, little to no coordination or collaboration exists among these three offices. Many researchers appearing before the Commission found it difficult to identify recent collaborative activities of these three offices despite the fact all three share the same floor of the same building and operate within the same federal agency under a single assistant secretary. The Commission encourages each OSERS office to work more closely together and foster a community of scholars working in tandem across the research activities occurring within OSERS.
The Commission was able to identify several examples of positive and productive interactions of OSEP research programs with other federal agencies that address research on people with disabilities. More than any other agency, OSEP has advocated including people with disabilities in the research and other activities, such as the Bureau of Census and national surveys. It is critical that federal agencies work together to ensure that common priorities are addressed and that duplication of effort is avoided. In this regard, some glaring examples of lack of interaction emerged. Despite the importance of research on scaling educational research, OSEP was not invited to participate in the Interagency Educational Research Initiative (IERI). This may reflect the perception of OSEP as an isolated research entity with limited funding. But, OSEP should be aware of these initiatives and pursue participation. Even if OSEP has limited funds to contribute, they should be at the IERI table to learn how scaling research occurs in other parts of the government and to provide input on how their research fits into scaling efforts.
In other instances, OSEP contributes to this perception.82 OSEP is considered to be the primary agency responsible for research on improving educational results for children with disabilities. Few agencies, or other offices within the Department of Education, construct their research efforts specifically to include children with disabilities or in a way that results for children with disabilities can be disaggregated. Although the term “educational” may be an important modifier, this should not be to the exclusion of other relevant education programs that address children with disabilities at other federal research agencies.
Particularly conspicuous is the absence of a relationship with the Center for Mothers and Children at the National Institute of Child Health and Human Development (NICHD). This Center includes the Mental Retardation/Developmental Disabilities Research Branch, which funds 15 national MR/DD research centers as well as significant research programs in autism and other disabilities. The Center includes the research on reading and reading disabilities by the Child Development and Behavior Branch of NICHD, which is also highly relevant to OSEP. There is a large program involving children with behavior disorders, but no systematic relationships with National Institute of Mental Health, where both organizations would benefit from mutual research efforts on the prevention of behavior disorders in children.
OSEP should systematically seek relationships and opportunities for interactions with and joint funding of its priorities with other federal research agencies. Similar expectations should be established for these agencies to interact with OSEP. People with disabilities should be included in all federal research programs whenever feasible and OSEP should continue to work towards this goal.
Invest in Long-Term Research Priorities
Inevitably, OSEP is faced with converging constituencies and a finite level of resources. A focus on priorities resulting in research activities with a potential for large-scale implementation and sustainability will more effectively combine resources for the maximum result. Furthermore, we recognize the importance of directing funds to support our recommendations. Thus, we find it important to continue to support investments in IDEA Part D research activities and to appropriately balance federal spending on research with spending on IDEA State Grants. IDEA Parts B and C must be guided by research, and continued investments are required to support foundational research to validate systems and programs that will facilitate bringing those findings to national scale and sustain their use over time.
Create a Community of Scholars within OSEP
The Commission recommends OSERS, and more specifically OSEP, increase the number of research scholars within its organization so a culture of scientific rigor can be supported and sustained. OSEP has too few seasoned researchers with the skills required to relate the effects of research to long-term improvement in results for children with disabilities. We support the idea of creating fellowship opportunities for those with newly granted doctorates and for senior researchers to serve within OSEP to increase the interactions between those truly conducting research and those administering research at the federal level.83
“Unless the broad array of issues related to scalability and sustainability are deliberately and aggressively addressed, the lofty vision and goals inherent in the No Child Left Behind Act will not be realized.”
—Donald D. Deshler, Ph.D.
A growth in research skill and competence must occur at OSEP. The intellectual capital of the agency is a cornerstone of future success. OSERS must be provided the flexibility to change the present focus to one defined by academic stimulation. This comports with the goal, supported by the Commission, of building infrastructures in federal education agencies that promote collaboration among researchers, policy makers and practitioners.84
Improve Development and Dissemination of Research Findings
Bridging the gap between research and practice will be a continuing challenge. Practices must continue to improve so children, their parents and their teachers have access to effective practices and instructional methods. A disturbing finding by this Commission is that we do not yet know the best methods of research dissemination. We have no research to show which methods prove more effective in reaching consumers. As a result, we can now make only three recommendations:
1. The Department of Education and other appropriate agencies should undertake systematic research efforts on effective dissemination practices and systematic efforts to bring to scale practices that are identified as effective. Both of these activities should be focused on speeding dissemination. Absent such reliable information, we support building a bridge to resolve much of the current gap between valuable research findings and those who use knowledge.
2. OSEP should be given funds to develop two missing components of its dissemination activities. One is a synthesis center; the other involves scaling centers focused on implementation of major innovations. These efforts should parallel and enhance the efforts of the Interagency Educational Research Initiative.
3. Federal law should be amended to address the federal government’s primary means of development of research and technical assistance—its regional education laboratories (RELs) funded under the U.S. Department of Education’s Office of Educational Research and Improvement, and its special education regional resource centers (RRCs) administered by OSEP. These institutions should both be obligated to improve their responsiveness to state-identified needs. The Commission recommends that RELs be obligated to include special education practices within the scope of their work. RRCs should be obligated to work closely with RELs or possibly be merged with them. RRCs and RELs should be held to more rigorous performance standards in technical assistance and research activities. The Department should also focus on ensuring that the quality of these programs is more uniform, so that the least of them rise to the quality of the best of them.
“As a state superintendent, I know how important it is to receive accurate information based on quality research findings to better serve children with disabilities. Federal research activities must provide the ability to scale on local and state levels.”
—Commissioner Nancy Grasmick, Ph.D.
The National Information Center for Children and Youth with Disabilities (NICHCY) should be restructured to address multiple topics with aggressive, strategic dissemination. It should link to empirically validated practices developed not only in OSEP but also in other federal research agencies. Materials should be largely devoted to the dissemination of scientifically based principles derived from empirical syntheses of rigorous research. There should be less emphasis on distributing information about the law and how to comply with it, and more emphasis on “what works” and outcomes.
The Commission found little value in continued support for the Educational Resources Information Center (ERIC) Clearinghouse, which is jointly sponsored by OSEP and OERI, given the wide availability of other library-based databases and search engines. The material is largely redundant.
State Improvement Grants, Personnel Preparation Grants and Parent Training and Information Centers should be linked with the research efforts and conceptualized as opportunities to either prepare teachers and researchers for careers involving people with disabilities based on scientifically based practices or opportunities to disseminate scientifically based information on research-based practices to parents and schools. Presently, these activities are poorly linked to research. IDEA should be amended to allow the Secretary to devote a percentage of each program’s respective budget to program evaluation and research on how to best establish these types of programs.
The Model Demonstration Projects should be expected to generate peer-reviewed publications and others demonstrating that they are research-oriented. At times, it was unclear whether the purpose of these projects was research or dissemination.
The Importance of Institutions of Higher Education in the Research Process
We cannot overlook the crucial importance of our nation’s institutions of higher education as partners in the production of research and as instruments of effective information dissemination, not only to training future educators, researchers and related services professionals but also to state and local education agencies. Special education research and related research comes primarily from our nation’s colleges and universities. Overall, investing in institutions of higher education must be made to expand and strengthen the special education research capacity by:
1) Ensuring the production of more doctorates in special education;
2) Providing incentives to doctorates (perhaps through post-doctoral fellowships) to do research in higher education; and
3) Developing more research institutes that address core questions at great depth over a long period of time, e.g., the relationship between teacher quality and student achievement.
We are also concerned with the increasing need for individuals trained at the doctoral level in special education and related services. Today, there are not enough individuals who hold doctorates in special education to adequately train special educators or to conduct the research needed to advance services for children with disabilities. This shortage in special education faculty requires our attention and leads us to suggest that specific efforts must be made to encourage talented special educators to pursue doctorates in special education. Incentives for prospective special educators to enter doctoral programs should include graduate fellowships or other stipends similar to those offered by the Rehabilitation Services Administration to train vocational rehabilitation counselors. The need for highly trained special educators we discuss in our Teacher and Administrator Preparation section of this report cannot be met if our nation’s colleges and universities have insufficient highly qualified doctoral level faculty to train students.
The recommendations set forth here reflect our belief that improving research is integral to improving our teacher training programs. Colleges and universities, Centers for Excellence in developmental disabilities and other university-affiliated programs are essential partners in supporting continued improvements in special education.
The Importance of Research in the Implementation of IDEA
It is recommended that OSERS collect and analyze data, which can inform the Department of Education and the public about the relationship between factors relevant to the implementation of IDEA and student outcomes and results. These factors include, for example, student achievement and learning, and post-school outcomes.
The figure on page 66 provides a schematic of a proposed research-to-demonstration-to-dissemination model that may be helpful in thinking about the relationship of research to practice. This figure shows a continuum from practice-oriented research in the field-initiated program that moves into model dissemination projects and then to large scaling centers that bridge the gap between research and dissemination. On the dissemination side, there are RRCs and NICHCY. Intermediate is the synthesis center. Presently, OSEP funds directed activities at a 2:1 ratio over field-initiated research. The field-initiated component should be increased so that it is comparable with the model demonstration projects, which are uneven in quality. Field-initiated research should be oriented to new ideas, which are vital to innovation, while model demonstration projects should be research-oriented but focused on initial applications in a manageable number of schools. OSEP should fund fewer of these activities and focus them on small-scale implementation of promising practices or synthesis of research around key areas. Scaling centers would need large budgets and would be oriented towards implementation of the most significant findings; rarely would more than one to three be in operation at any point in time. The synthesis center would monitor all research findings at OSEP and other areas, feeding these results to the dissemination components. To facilitate development of this capacity, a significant increase in Part D funding is strongly encouraged.
Organizational Model for Research to Dissemination
Special Education Research Agenda
The Commission recommends that the federal government undertake and support research in the following areas, as requested by the President in section 3(b)(3) of Executive Order 13227:
1. Implementation of models for response to intervention
2. Implementation of continuous progress monitoring; additional research extending these methods to middle and high school, significantly disabled students, and in areas beyond reading (such as math and behavior); development of national norms.
3. Development of assessment methods that are based on universal assessment; alternate assessments; evaluation of methods for assessing adequate yearly progress for students with disabilities.
4. Parent-based early childhood interventions linked with NICHD research on the development of and intervention with high-risk infants and preschoolers.
5. Scalability and sustainability research linked with IERI.
6. Learning disabilities in older children and in areas other than reading linked to NICHD research.
7. Development and adaptation of instructional methods for children with low-incidence disabilities linked to NICHD research.
8. Costs of special education, including data collection in the following areas:
a. Data collection and analysis of true excess special education costs;
b. Use of a census-based formula for distribution of special education funds;
c. Determination of the costs and necessary resources for student achievement of identifiable outcomes and results;
d. Influence of eligibility parameters and uses of other federal funds for high-need students;
e. Impact of state special education high-need reimbursement models and risk management pools on service delivery models and settings in special education;
f. Influence of pooling Part C and Section 619 funds on early intervention success; and
g. Impact of fiscal reforms in special education on the general education program.
9. Adaptation of instructional methods that promote inclusive educational practices.
10. Instructional methods for students with autism linked to NICHD/CDC.
11. Evaluation of parent training and information programs.
12. Professional development research including:
a. Identification of the critical factors in personnel preparation that improve student learning and achievement in schools, and the instructional variables that improve student learning;
b. Determination of whether teacher certification or years of expertise contributes to student achievement;
c. Identification of what teacher characteristics affect student achievement, and what factors have a strong effect in producing student achievement;
d. Identification of effective strategies to address the current personnel shortage; and
e. Determination of the best means of professional development for general education and special education teachers that lead to effective implementation of research-based practices.
13. Identification of the factors that help students with disabilities make the transition into college, and model programs utilizing that scientifically based research.
14. Collection and analysis of data on students served under “504 Plans,” including the effectiveness of 504 Plans and the number of 504 Plan children with disabilities gaining employment or moving onto higher education.
15. Collection and analysis of data on due process and dispute resolution.