Guidelines and Materials for Submitting
Educational Technology Programs
for Review

• Background and Purpose of the Expert Panel
• How to Complete a Program Submission
• Evaluation Criteria
• Application to Be Completed

Background and Purpose of the Expert Panel

What Is the Expert Panel?

The Educational Research, Development, Dissemination and Improvement Act of 1994 directed the Assistant Secretary for the Office of Educational Research and Improvement (OERI) to establish "panels of appropriate qualified experts and practitioners" to evaluate educational programs and recommend to the Secretary of Education those programs that should be designated as promising or exemplary. The Educational Technology Expert Panel (the "Expert Panel") is one of those panels. Other Expert Panels have been convened for Mathematics and Science Education; Gender Equity; and Safe, Disciplined, and Drug-Free Schools.

The purpose of the Educational Technology Expert Panel is to develop a valid and viable process for designating promising and exemplary programs using learning technologies and identifying the conditions under which exemplary practices develop. This will help educators throughout the country make better-informed decisions in their ongoing efforts to improve the quality of teaching and learning supported by technology. Specific responsibilities of the Expert Panel include developing, revising, and refining the criteria and review process; identifying, selecting, and training reviewers and review panels; making recommendations to the Secretary for designating programs as promising and exemplary; and contributing to the broad and effective dissemination of information about designated programs.

The identification of promising and exemplary technology-based educational programs is one outcome of this effort. At least as important, however, is the development and dissemination of (1) criteria for excellence in the use of educational technology and (2) sophisticated strategies for adapting successful implementations of educational technology programs from one educational site to another. The Expert Panel's application materials provide conceptual frameworks for these important issues.

What Is the Vision for Technology in Education?

Our educational system must integrate current and future information and communications technologies into everyday teaching and learning. The challenge is to create programs that capture the opportunities promised by emerging technologies in ways that are affordable, sustainable, and scalable. Learning technologies are effective only when treated as one component in implementation strategies that also encompass curricular reform, sophisticated and multiple assessments, effective professional development, well-maintained technology infrastructures and support systems, attention to equity, and the restructuring of organizational practices, budgets, and policies.

An exemplary technology-based program will incorporate all of these dimensions. A promising one may incorporate some of them and will include plans to achieve the remainder. The ultimate goal of the linked elements of any technology program designated by the Secretary is increased student learning.

Why an Award Now?

Educational technology is a constantly evolving field. In recent years, schools and communities have made significant investments to provide their teachers and students with appropriate information and learning technologies. The difficulty many educators face is how best to adapt current structures to achieve the potential of these emerging technological tools.

The Secretary and Expert Panel expect the promising and exemplary programs to serve as models to a wide variety of institutions. The Department of Education intends to publish descriptions of the selected programs on the OERI website to inform and encourage the sharing of successful models. Designated programs should be prepared to host a variety of visiting educators and to provide advice and demonstrations to help schools and related sites shape and support the work of effective teaching and learning.

The application process asks for information that can serve as a rich dissemination resource for other groups wishing to adapt and implement that program. The U.S. Department of Education's six Regional Technology in Education Consortia (R*TECs) will also assist with the dissemination of exemplary and promising programs.

Programs can benefit from the Secretary's designation through media, news coverage, and professional networking opportunities, including potential invitations to present at professional conferences and other educational settings. A program's designation as promising or exemplary is also likely to enhance its competitiveness for future funding opportunities.

What Kind of Programs Will Be Recognized?

For this award, the term educational technology encompasses a variety of electronic tools, media, and environments that can be used to enhance learning, foster creativity, stimulate communication, encourage collaboration, and engage in the continuous development and application of knowledge and skills.

A program means the implementation of a set of interrelated strategies and activities directed toward increased learning. Eligible programs will depend on technology to achieve their goals. A program includes assessment that demonstrates its impact on participants and upon a wider organizational structure (i.e., a school or cluster of schools, a district, community organization, partnership, or other distributed system).

To be designated as exemplary or promising, programs must go beyond fine-tuning, automating, or moderately enhancing conventional educational methods: they must demonstrate the capacity to effect substantial improvements in PreK-12 education and to understand the conditions necessary to their success. These improvements can be achieved through programs focused directly on student learning, or through the significant ancillary activities of teacher professional training, increasing equity, and/or organizational reform.

A wide variety of applications is invited. The Secretary intends to identify and honor innovative programs that demonstrate how using technology can support:

• Teaching and learning in all curricular areas or across disciplines
• Advancing the professionalism of educators
• PreK-12 learning supported by community-based or business organizations
• Reorganizing of schools and other settings to enhance educational effectiveness
• Improving educational access and equity
• Preparing students for 21st-century work and citizenship.

In sum, the Secretary wishes to recognize, in particular, comprehensive technology-based innovative programs that lead to improvements in the quality of teaching and learning taking place in our nation's schools.

How Will Applications Be Reviewed?

Programs submitted to the Expert Panel will be evaluated based on four categories of criteria: (1) Quality of Program; (2) Educational Significance; (3) Evidence of Effectiveness; and (4) Usefulness to Others.

Submissions will be evaluated by trained Quality Review Panel (QRP) teams, drawn from a pool of qualified practitioners and researchers with technology and content expertise as well as classroom teaching experience. In addition, programs that are rated highly by the QRP will also be reviewed by an Impact Review Panel (IRP) - constituted of national experts in evaluation design and analysis - which will focus on the persuasiveness of the Evidence of Effectiveness. The Expert Panel will review the evaluation ratings and comments of both the review teams as well as the materials submitted, to determine which programs to recommend to the Secretary as promising or exemplary.

Educational Technology programs will be reviewed in the Fall of 1999, and Panel recommendations will be made to the Secretary in early Winter. The Secretary will announce the final designated programs.

What Characteristics Will Be Judged?

Candidate programs must demonstrate persuasively that:

• The program addresses significant educational issues and improves PreK-12 learning;
• The program makes possible educational gains that cannot be achieved without the use of technologies; and
• The program can serve as a model for other educational institutions because it is sustainable, adaptable, and scalable.

Who Can Apply?

The lead applicant must be either:

• A local educational agency (LEA) or a school district on behalf of an individual school or group of schools;
• A for-profit organization or institution in collaboration with at least one LEA;
• A nonprofit organization - e.g., state education agency (SEA), institution of higher education (IHE) in collaboration with at least one LEA; or an Intermediate Education Unit, or consortium of school districts; or
• A private school or group of private schools.

The application is structured to help applicants make a convincing case for the value and effectiveness of their program with respect to six criteria (see below). Clarity and compelling evidence are essential throughout. Because the elements are interrelated and the evidence required is rigorous, it is recommended that applicants review all six criteria before beginning to complete the application.

In keeping with the nature of educational technology, all submissions are to be made electronically. Detailed instructions for completing a program submission are provided in the following section. If a program is designated as promising or exemplary, a description of the program, subject to approval by the applicant, will be featured on the U.S. Department of Education's website for dissemination purposes. If a program is not selected as promising or exemplary, no information about it will be made public.

What Materials Should Be Submitted?

Applicants are to complete the Web application form outlined in the following section. They will address each of the six criteria listed below. In addition, the applicant will submit an abstract; a budget outline summarizing the actual program costs, as well as the infrastructure, training, and support costs necessary to implement and sustain the program; and demographic and funding information. Lastly, the applicant will print out the Certificate of Accuracy and Completeness (the final item of this Application), then sign and mail the hard copy of the certificate to the Expert Panel (c/o RMC Research, address below).

Submit the completed application through the web site at URL <http://www.ed.gov/offices/OERI/ORAD/LTD/final/security/>.

The Educational Technology Expert Panel will evaluate programs as presented in the application. Certain additional materials may be submitted as well.

• Evaluation reports: Reports or summaries of evidence supporting the claims of effectiveness may be sent in hard copy. Such reports may include evaluation studies, published articles, or works in progress making claims about the program's effectiveness; do not, however, include questionnaires or raw data. If submitting evaluation reports, send three copies, each labeled with a printout of the completed Cover Sheet, to the address below. Note that these items will not be returned.

• Supplemental materials: In addition, up to three programmatic items or work products documenting claims in the application - such as websites, CD-ROMs, videotapes, and other multimedia materials - may be submitted. A URL may be cited in your response to the criteria where it is most appropriate. If submitting actual materials, send one copy, clearly labeled with a printout of the completed Cover Sheet, as well as a reference to the relevant section of the application, to the address below. The Panel may consult these materials if it seeks further information regarding an application. As with the evaluation reports, no supplemental work products will be returned, so do not send unique pieces.

Send a hard copy of the Certificate of Accuracy and Completeness (Required) Evaluation Reports (Optional) Supplemental Materials (Optional) (A Cover Sheet must accompany the above items) to:

Educational Technology Expert Panel c/o RMC Research Corporation 1000 Market Street, Building 2 Portsmouth, N.H. 03801 Phone: 603-422-8888

Deadline for Submission

To be reviewed in Fall 1999, submissions must be received on the website, and the Certificate of Accuracy and Completeness as well as any supporting materials at RMC Research, by 5:00 p.m. EDT, September l, 1999.

The submission deadline will not be extended for any reason; you are therefore encouraged to avoid last-minute entries. On-line submissions will be accepted starting on July 15.

Information Required for Program Application

When ready to complete the application form, access the URL listed above and click on the Application button. Essential information is repeated on the Web application itself.

1. Cover Sheet including the Title of Program and Name of Primary Contact Person.

2. Program Profile and Demographic Data.

3. Program Abstract. Briefly summarize the program, program goals, population served, and outcomes (250 words).

4. Program Costs. Indicate the total start-up and ongoing costs for implementing the program and give a breakdown (e.g., personnel, training, materials and supplies, hardware, software, technical support, replacement or upgraded hardware or software, etc.). Then explain briefly how these costs are related to major program activities (maximum of 250 words).

5. Meeting the Evaluation Criteria. Address each of the six criteria as specified, and justify the merit of the program. Responses should acknowledge the program's weaknesses as well as strengths; specify how any perceived weaknesses were overcome or will likely be overcome (i.e., lessons learned). Use the bulleted indicators under each criterion as guidelines in your response.

6. Certificate of Accuracy and Completeness.

Review Procedure

Program submissions will be judged against six criteria, each with several indicators or guidelines. Reviewers will use the 5-point rating scale below to indicate the degree to which each criterion is met.

1   Unsatisfactory or Incomplete
2   Fair
3   Good
4   Very Good
5   Excellent

A rating of 2, for example, means that the program is weak on a majority of indicators. A rating of 3 means there is evidence that a majority of the indicators for a criterion are met.

Evaluation Criteria

The Expert Panel is seeking technology-dependent programs that will help improve PreK-12 teaching and learning in the United States. Technology by itself, however, cannot generate the desired educational outcomes. The Panel members have found that PreK-12 learning is most effective and lasting when supported by attention to several ancillary but interconnected areas. The most important of these are equity (advancement of all student populations) and organizational change (e.g., allocation of time and resources for teacher professional development).

The diagram below presents the interrelatedness of the six criteria upon which programs will be reviewed. Recognized programs will have significant educational goals that result in complex learning supported by technology. These programs will also promote organizational change as well as greater equity and educational excellence for all students. Such programs should be able to demonstrate persuasively their effectiveness regarding these outcomes and be useful and adaptable in other school settings. Applicants can refer to this diagram in conceptualizing how their programs meet these six criteria.

Note: The term learners can refer to PreK-12 students, educators, or parents.

1. Quality of Program (maximum of 2,500 words for Criterion 1)

   Criterion 1. The program addresses an important educational issue or issues and articulates its goals and design clearly.

   • The educational goals are significant.

   • The program design is thoughtful and supported by research.

   • The program description is clear and complete.

   Please describe the program in detail. Readers should be able to understand the overall program, as well as what participants actually do, sufficiently well to explain the program to others. Include the following items:

   a. Need or problem the program addresses and how it relates to teaching and learning in PreK-12 schools.

   b. Program goals.

   c. Technology used and how it helps to achieve the program's goals.

   d. Subject population(s); include ethnic, racial, socioeconomic, and gender percentages. Include the size of any special populations served (e.g., ESL, AP biology students, students with disabilities).

   e. Content and learning goals.

   f. Program design (structure and components).

   g. Professional development provided as part of the program.

   h. Overall size and maintenance costs (funding and staff requirements, number of people in target population).

   i. Key learning activities for participants.

   j. Assessment(s) used to determine the program's efficacy and achievements.

   k. Keys to the program's success.

   l. A specific, concrete example that best captures the changes achieved by this program.

2. Educational Significance (maximum of 2,500 words for Criteria 2, 3, and 4 combined)

   The Expert Panel considers the following three areas ? learning, equity, organizational change ? essential to fulfilling the promise of educational technology. A sound program must address all three, and all three must be shown to have impact on or linkage to PreK-12 student learning.

   Criterion 2. The program develops complex learning and thinking skills for its target audience.

   If the target audience is other than PreK-12 students, the applicant should articulate the program's goals and their connection to student learning. If the target audience is PreK-12 students, the indicators might include one or more of the objectives below:
   • The program increases students' in-depth understanding and competence in at least one content discipline.

   • The program develops the habits of lifelong learning (e.g., the ability to collaborate, to direct one's own learning, to solve problems, to communicate ideas clearly, to think flexibly and critically).

   • The program helps students become proficient and critical consumers and producers of educational technology.

   • The program includes preparation for entering a technology-infused work place.

   Criterion 3. The program contributes to educational excellence for all.

   • The program conveys high expectations for all learners.

   • The program responds to the diverse needs of varied populations of learners.

   • The program includes active outreach and partnerships to encourage broad participation.

   • The program increases the participation or achievement of underserved learners so that the gap between this group and other categories of students diminishes.

   Criterion 4. The program promotes coherent organizational change.

   • The program reflects a vision of educational reform consistent with disciplinary content standards, recommendations from national commissions, findings from educational research, and documented best practices.

   • Policies, funding, and practice are aligned to support sustainable change.

   • Through partnerships and professional development, the program builds human capacity to accomplish its goals (e.g., allocates time for teachers' and administrators' collaboration and planning).

   • The program increases the educational involvement of parents, professional groups, and communities.

3. Evidence of Effectiveness (include as much material as necessary to make a persuasive case)

   Criterion 5. The program has rigorous, measurable evidence for its achievements for at least one criterion among Criteria 2, 3, and 4 (learning, equity, organizational change ).

   Describe clearly the major achievements of the program and the evaluation methods used; provide convincing evidence of the program's claims. Do not omit mixed or negative results. (As discussed in "What Materials Should Be Submitted?," evaluation reports may be mailed as hard copy.)

   In addition to the narrative description, summarize your response to Criterion 5 in the Claims of Program Effectiveness Chart (in Appendix A). Appendix A also provides additional information about evidence.

   Valid evidence will meet generally accepted standards in the field and may include:

   • One or more comparison groups

   • Quantified validation by an external authority

   • A formal evaluation

   • A quantified demonstration of positive change among participants as a result of the program (e.g., increased parental involvement in school governance; diminished gaps in achievement between groups; increased enrollment in rigorous mathematics courses or graduation rates among subject populations; changes in the base funding and requirements for professional development)

   • An in-depth, qualitative analysis of change among participants as a result of the program (e.g., case studies, ethnographies, principled analysis of observations and interviews).

   To meet this criterion, programs must provide clear and rigorous evidence that the program has resulted in important changes in the behavior or performance of its intended population; attitudinal changes will be considered supplementary rather than primary. Evidence of improvement can be demonstrated by a variety of thoughtful means and is not limited to higher scores on standardized tests. In general, multiple forms of supporting evidence will make a stronger case than a single form of evidence.

   For example, a program ("1-2-3 MATH") that claims students learn to apply complex mathematical concepts to real-world situations as a result of this technology-dependent program would do well to present analyses of its assessment of students' mathematical work over time and in comparison to groups that did not participate in the program. Student performance on standardized tests in mathematics would also add credibility to the claim. Surveys or interviews with the students and/or their parents indicating that students are enthusiastic about the program and that they connect more mathematical ideas to real-life situations will contribute to but not substantiate the claims.

   Attitudinal surveys or anecdotal stories by themselves, however, are not sufficient evidence. Testimonials will not be treated as credible evidence unless they cite demonstrated accomplishments made possible by the program. For example, praise or commendation of learners by a local official will not be considered evidence of a program's effectiveness. If that same official, however, attests that the students' analysis of rush-hour traffic flow provided the basis for the town council's reconsidering traffic patterns, that would be considered relevant evidence.

   If the 1-2-3 MATH program chooses to demonstrate effectiveness with respect to equity, it could show that underserved students are significantly benefiting from the program. It might, for example, use classroom interview and observational studies to illustrate how curriculum and pedagogy have been adapted to the populations served (e.g., problems and projects are directly connected to the students' cultural and community context). The program could also provide data affirming its effect on the behavior of the target population (e.g., improved attendance rates, decreased dropout rates, increase in number of completed homework assignments, etc.) in comparison to a control group.

4. Usefulness to Others (maximum of 750 words)

   Programs may or may not have expanded beyond their original targets of use; however, to be designated as promising or exemplary, programs must convince reviewers that they are not unique exemplars. To do so, they should address the following criterion and its indicators.

   Criterion 6. The program is adaptable for use in multiple contexts.

   • The program's technology requirements are easily available to potential users.

   • The program is cost-effective relative to its benefits.

   • After its initial implementation, the program is sustainable with existing resources (i.e., does not require extraordinary/unreasonable time, effort, or funding), and scalable (i.e., can naturally expand its scope to several teachers, multiple grade levels/subjects/sites, different disciplines, etc.).

   • The program is adaptable to a range of educational settings with learners similar to the intended population.

   • The program provides clear and detailed guidelines about the conditions required for its successful implementation.

Application to be Completed

-###-